Better Backend DX: Fastify + ESBuild = ⚡️

Hi! I'm David Peng👋, you can find me on Twitter: @davipon.

I'm currently working on an internal legacy project which I mentioned in my old blog post: "Supercharge Web DX in Svelte Way".

I built a back-end service from scratch in the past few months, part of the monolithic to microservice migration. It was also my first time making all technical decisions like:

• Which framework to choose? Express, Koa, Fastify, or NestJS?
• DB choice? SQL or NoSQL? Use ORM/ ODM or not?
• Understand networking like transport protocols (UDP, TCP, and TLS), application protocols (HTTP/1.1, HTTP/2)
• Do we need a load balancer/ reverse proxy? Apache, Nginx, HA Proxy, or Envoy?
• Containerize the application or not? Will we need orchestration in the future?
• How to build & test in the development, stage, and production environment?
• What about the CI/ CD pipeline? DevSecOps?
• 🤯

It's intimidating😅, but I learned so much by getting my hands dirty. Even though we haven't finished the migration yet, I still want to write them down and start a new blog series.

I'll share my experience and thought process of building a production-ready backend service with a better DX.

There will be four parts (I might change the title in the future 🙃):

1. Better Backend DX using Fastify & ESBuild
2. Build a REST API using MongoDB
3. Build a minimal docker image
4. Add Envoy as a sidecar proxy

Let's start with the first one!

Better Backend DX (Developer Experience): Fastify + ESBuild = ⚡️

Below are my two beliefs about solving a real problem:

Build things that just work.

Better DX leads to better UX and better products.

When building a SvelteKit app, it's full of joy with a great DX. (Kudos👏 to the community and modern toolings.) But when it comes to JavaScript backend development, the DX is still like in the stone age.

How is the general backend DX?

Just my two cents:

• Lacks modern toolings like hot module replacement
• Restrictions of Node.js - immature ESM support leads to extra setup or compilation to cjs
• tsc/ ts-node + nodemon still slow as hell
• No interest in bundling production code (hard to debug? we don't care because it doesn't ship to clients? 🤷)

I couldn't stop thinking about these when I started to develop a pure backend. It seems to me that there is a massive gap between the frontend & backend world in terms of toolings and DX.

Let's say you want to build a node app using TypeScript, and you'd probably use tsc, ts-node, concurrently, or nodemon to:

1. Watch and compile your project from ts to js
2. Start the server

It works, but the problem is that you'll have a broken/ long feedback loop. It recompiles the whole project whenever a file changes, and tsc is way too slow:

ref: Benchmarking esbuild, swc, tsc, and babel for React/JSX projects

That's how I feel! I missed Hot Module Replacement and Vite goodness... wait, can we use it?

I used Vite in my React project, and SvelteKit also uses Vite under the hood. So I wondered if it's possible to use Vite in the backend development?

Vite for Node apps?

Yes, there is a Vite plugin: vite-plugin-node which leverages Vite's SSR mode to bring HMR to the Node dev server and also provides adapters for Express, Koa, and Fastify.

I also found a nice vite-fastify-boilerplate which uses the plugin & Vitest.

After trying them out, the feedback loop was much shorter than before thanks to HMR & esbuild, but there are two problems:

1. Websockets don't seem to work, at least with Fastify #22
2. I Haven't found a way to bundle production code using vite-plugin-node. (Vite SSR mode doesn't support bundling built-in Node.js lib

I like to bundle/ minify the production code because I want to build a docker image that doesn't need to copy the huge node_modules to the final image. I'll explain in a future post 😉.

Let's step back and compare different approaches:

Hmm🧐, it seems I need to find an alternative approach, so I started to experiment with ESBuild.

Use esbuild as a compiler and backend bundler

If we don't use Vite, we can't have HMR in the dev server.
But we can still use esbuild, the compiler that Vite uses, to replace tsc.

esbuild is so fast that even if we use nodemon to monitor and recompile the entire app, we can still have a short feedback loop.

esbuild-node-tsc is one of the packages that use this approach.

You can also try ts-node + swc + nodemon. ref: https://typestrong.org/ts-node/docs/swc

To have a better backend DX, there are more factors to consider besides dev server and production bundling. Before we start to set up the fastiy + esbuild project, let's talk about Fastify.

Why did I choose Fastify?

Here are the reasons:

• Support TypeScript out of the box
• Baked-in validation using JSON-Schema
• Extensibility - hooks, plugins, and decorators
• Good DX, e.g., great logger using pino, rich plugin ecosystem.
• @fastify/autoload enables filesystem-based routes & plugins

Performance was not my primary consideration.

Both Express & Koa are fast enough for a small project. IMHO, It would be better to spend dev time optimizing and parallelizing backend calls and DB queries than speeding up a router by 2ms.

Let's build a simple fastify + esbuild app!

This section aims to build a scalable starter template with a better DX.

Here is the GitHub repo: github.com/davipon/fastify-esbuild

# Create a new project
mkdir fastify-esbuild
cd fastify-esbuild
pnpm init
# Install dependencies
pnpm add fastify fastify-plugin @fastify/autoload
# Install devDependencies
pnpm add -D typescript @types/node nodemon esbuild

Create a src folder and index.ts under src:

// src/index.ts
import Fastify from 'fastify'

const fastify = Fastify({
  logger: true
})
// Declare a route
fastify.get('/', function (request, reply) {
  reply.send({ hello: 'world' })
})
// Start the server
const start = async () => {
  try {
    await fastify.listen(3000)
  } catch (err) {
    fastify.log.error(err)
    process.exit(1)
  }
}
start()

We use nodemon to monitor & restart the dev server.
Let's create nodemon.json under your project root folder:

// nodemon.json
{
  "watch": ["src"],
  "ignore": ["src/**/*.test.ts"],
  "ext": "ts,mjs,js,json,graphql",
  "exec": "esbuild `find src \\( -name '*.ts' \\)` --platform=node --outdir=dist --format=cjs && node dist",
  "legacyWatch": true
}

The "exec" script does following:

1. Compile all ts files under src folder to cjs (CommonJS) format and output to the folder dist
2. Start the dev server with node dist (or node dist/index.js)

Caveat

find src \\( -name '*.ts' \\) is only available in Linux/ macOS.

Will provide an alternative solution for Windows later. (ref: https://github.com/evanw/esbuild/issues/976)

The reason to assign all files but not just the entry like src/index.ts in the esbuild script is that we need to use @fastify/autoload later.

Then add scripts in package.json:

...
"scripts": {
    "dev": "nodemon",
    "build": "rm -rf build && esbuild `find src \\( -name '*.ts' \\)` --platform=node --outdir=build --bundle",
    "start": "pnpm build && node build"
  },

Let's try pnpm dev. You should see something like this:

Enables filesystem-based routes & plugins using @fastify/autoload

First create a routes folder under src, and then create a root.ts under routes:

// src/routes/root.ts
import { FastifyPluginAsync } from 'fastify'

const root: FastifyPluginAsync = async (fastify, opts): Promise<void> => {
  fastify.get('/', async function (request, reply) {
    return { root: true }
  })
}

export default root

Then in your src/index.ts:

import Fastify from 'fastify'
import { join } from 'path'
import autoLoad from '@fastify/autoload'

const fastify = Fastify({
  logger: true
})

// Will load all routes under src/routes
fastify.register(autoLoad, {
  dir: join(__dirname, 'routes')
})

const start = async () => {
  try {
    await fastify.listen(3000)
  } catch (err) {
    fastify.log.error(err)
    process.exit(1)
  }
}
start()

Let's give it a try!

Awesome🥳! Now let's add more routes:

Create a examples folder under routes and create index.ts in it:

// src/routes/examples/index.ts
import { FastifyPluginAsync } from 'fastify'

const examples: FastifyPluginAsync = async (fastify, opts): Promise<void> => {
  fastify.get('/', async function (request, reply) {
    return 'Hi there!'
  })
}

export default examples

Here is the demo:

With @fastify/autoload, we can easily add plugins and routes. In the future, you might want to deploy some of those independently. It also provides an easy path to a microservice architecture.

You can see a basic structure with routes and plugins like this:

⚠️ Windows' solution for the esbuild script

Create esbuild.js under the project root folder.

const env = process.argv[2]
/* eslint-disable @typescript-eslint/no-var-requires */
/* eslint-disable no-undef */
const fs = require('fs')
const path = require('path')
const esbuild = require('esbuild')

let fileArray = []
const getFilesRecursively = (dir) => {
  const files = fs.readdirSync(dir)
  files.forEach((file) => {
    const filePath = path.join(dir, file)
    if (fs.statSync(filePath).isDirectory()) {
      getFilesRecursively(filePath)
    } else {
      fileArray.push(filePath)
    }
  })
}
getFilesRecursively('src')

const entryPoints = fileArray.filter((file) => file.endsWith('.ts'))

esbuild.build({
  entryPoints,
  logLevel: 'info',
  outdir: env === 'dev' ? 'dist' : 'build',
  bundle: env === 'dev' ? false : true,
  platform: 'node',
  format: 'cjs'
})

Replace esbuild script in both nodemon.json and package.json:

// nodemon.json
{
  ...
  "exec": "node esbuild.js dev && node dist",
}

// package.json
{
  ...
  "script": {
    "build": "rm -rf build && node esbuild.js",
  }
}

That's it! 🎉 This is Part 1.

Thank you for your reading!

You may notice that this fastify + esbuild is heavily inspired by modern web frameworks and toolings.

What motivated me to write this series was the frustration. You can find many resources and articles about modern tools and how they improve frontend DX, but only a few in the backend world talk about it.

In the frontend world, Vite's approach to leveraging ESM & HMR to tackle slow server start and bundling is fantastic. I hope we can have native backend/ Node.js support in the future.

In the next blog post, I'll share how we can use this fastify + esbuild template to:

• Build REST API using MongoDB
• Use json-schema-to-ts to validate data and stop typing twice
• Use native MongoDB driver instead of ODM like Mongoose

Kindly leave your comment and thoughts below!

See you in the next one!