Running Node.js on AWS serverless with Fargate

Last week I had a fun project to stand up a Node.js stack for integration testing. While I've run JavaScript in a Node.js runtime on AWS Lambda many times, this testing required a backend server with more complexity, such as long-lived connections and persistent state. A great option would be to build and run the app as a docker container, but the testing environment didn't have k8s setup. Enter AWS Fargate, a container runtime that doesn't require spinning up my own infrastructure! Fargate will manage the infrastructure and the networking, and automatically pull a container from a registry such as Elastic Container Registry or Dockerhub to run with minimal configuration required.

Of course for most real-world applications, such as web servers, running a container alone is not sufficient. There is also the need to accept HTTP requests from the internet, manage DNS records, and connect to backend databases. Fortunately AWS provides options for all of these needs, with a setup called http api private integration. In a nutshell, this allows for hosts running in a private VPC to expose HTTP API endpoints to the internet through a gateway. Now the container run by Fargate can use the same isolated VPC network as other backend services and databases, without exposing them all to the public internet!

I couldn't find an up-to-date example with Terraform, so I decided to make this one! There are more than a few parts to this stack - an API gateway, application load balancer, ECR container registry, ECS cluster with Fargate, SecretsManager variables, KMS key, and Cloudwatch logs.

The headline image of this post shows the architecture of AWS services being used. Additionally, the complete code for this example can be found in this github repository subdirectory

Forest Admin

I didn't have a spare node app sitting around, so I found Forest Admin. This is actually a cool product which provides the simplicity of dashboard tools like ActiveAdmin or Retool, but preserves the privacy of the data by having you self-host the backend. The backend exposes an API that is used by the frontend client, i.e. your browser, so data doesn't need to move through Forest Admin's servers. Here's a nice graphic to visualize how this works:

Forest Admin releases their backend with a variety of frameworks - we've chosen Node.js with Express for HTTP and Sequelize for SQL ORM.

How this works

We will use Terraform to build and upload a docker container to ECR, in addition to configuring all of the AWS resources. This means I can make code or infrastructure changes, and deploy everything with a single terraform apply command.

Note: I had an existing VPC and RDS PostgreSQL database running in the testing environment, so they are excluded from this example.

I wanted to accomplish the following in this project:

• Build and distribute a Node.js app as a docker container
• Create production-grade architecture - use a load balancer for scaling, inject secrets securely as ENV variables, and aggregate logs for debugging
• Run the container in a private network alongside other backend servers, such as RDS
• Expose an HTTP API to the public Internet running in the container with node.js Express
• Use Terraform to build and deploy

The key pieces to getting this all to work are:

• API Gateway v2 can create a "VPC link", to forward HTTP traffic from the internet to a private host in a VPC. This allows us to keep the VPC subnets network private, i.e. not exposed to the internet.
• Elastic Container Service can manage a load balancer to distribute requests to Fargate tasks. This lets us scale up and down the number of independent containers running, and eliminates the need to register task IP addresses or health checks in the load balancer.
• Terraform can run a docker build command on our localhost and upload the container to Elastic Container Registry. While a mature product would have a CI/CD process set up for this, we can skip all that for smaller projects like this one.

Docker file and app.js

Created a minimal Dockerfile for this project

FROM node:16

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
COPY yarn*.lock ./

RUN yarn install --immutable

# Bundle app source
COPY . .

EXPOSE 3000
CMD [ "node", "src/app.js" ]

app.js is a bit more interesting, we bind to 0.0.0.0 as the host to expose through the container. There is a health check endpoint configured for the root path /, so the load balancer can register a task as healthy.

require("dotenv").config();
const name = "forest-admin";
const express = require("express");

// Constants
const PORT = 3000;
// Important to bind to all hosts to run in a container
const HOST = "0.0.0.0";

// App
const app = express();
// Healthcheck route for ALB
app.get("/", (req, res) => {
  res.send("ping");
});

// Retrieve your sequelize instance
const { createAgent } = require("@forestadmin/agent");
const {
  createSequelizeDataSource,
} = require("@forestadmin/datasource-sequelize");

const { Sequelize } = require('sequelize');

const sequelize = new Sequelize(
  process.env.POSTGRESQL_URL
);
require("./models")(sequelize);

// Create your Forest Admin agent
// This must be called BEFORE all other middleware on the app
createAgent({
  ...config params...
})
  // Create your Sequelize datasource
  .addDataSource(createSequelizeDataSource(sequelize))
  // Replace "myExpressApp" by your Express application
  .mountOnExpress(app)
  .start();

...

app.listen(PORT, HOST, () => { });

Building the Docker container

We can use Terraform to run a docker build command on our localhost using a null_resource. After building, it will push the container to an ECR repository also managed up by Terraform. It can take advantage of an explicit triggers { ... } block to only rebuild when source files change. Docker and AWS cli v2 are prerequisites for this null_resource to execute correctly.

resource "aws_ecr_repository" "task_repository" {
  name         = "${var.service_name}-repository"
  force_delete = true
}

resource "null_resource" "task_ecr_image_builder" {
  triggers = {
    docker_file    = filesha256("${local.root_dir}/Dockerfile")
    package_file   = filesha256("${local.root_dir}/package.json")
    yarn_lock_file = filesha256("${local.root_dir}/yarn.lock")
    src_dir        = sha256(join("", [for f in fileset("${local.root_dir}/src", "**.js") : filesha256("${local.root_dir}/src/${f}")]))
  }

  provisioner "local-exec" {
    working_dir = local.root_dir
    interpreter = ["/bin/bash", "-c"]
    command     = <<-EOT
      aws ecr get-login-password --region ${var.region} --profile ${var.aws_profile} | docker login --username AWS --password-stdin ${local.aws_account_id}.dkr.ecr.${var.region}.amazonaws.com
      docker image build -t ${aws_ecr_repository.task_repository.repository_url}:latest .
      docker push ${aws_ecr_repository.task_repository.repository_url}:latest
    EOT
  }
}

data "aws_ecr_image" "task_image" {
  depends_on = [
    null_resource.task_ecr_image_builder
  ]

  repository_name = "${var.service_name}-repository"
  image_tag       = "latest"
}

Terraform

(Please reference the main.tf file in the repository here for a complete list of resources. Some are excluded here for brevity)

Keys, Secrets and Logs

We store sensitive variables as a JSON object in AWS SecretsManager. These variables can be referenced in the Fargate docker task defition, and will be securely injected as environment variables to the docker container at runtime. For input, we mark the terraform variables with sensitive = true to ensure they don't end up in output or state files. Also logs.

locals {
  ...
  task_secrets = {
    FOREST_ENV_SECRET  = "${var.forest_env_secret}"
    FOREST_AUTH_SECRET = "${var.forest_auth_secret}"
    POSTGRESQL_URL     = "${var.postgresql_url}"
  }
}

resource "aws_kms_key" "kms_key" {
  description             = "KMS key for encrypting secret environment variables and cloudwatch logs from the ECS cluster"
  deletion_window_in_days = 7
}

resource "aws_kms_alias" "nodejs_alias" {
  name          = "alias/ecs/${var.service_name}"
  target_key_id = aws_kms_key.kms_key.key_id
}

resource "aws_secretsmanager_secret" "task_secrets" {
  name_prefix = "/ecs/${var.service_name}"
  kms_key_id  = aws_kms_key.kms_key.id
}

resource "aws_secretsmanager_secret_version" "task_secrets" {
  secret_id     = aws_secretsmanager_secret.task_secrets.id
  secret_string = jsonencode(local.task_secrets)
}

resource "aws_cloudwatch_log_group" "nodejs_log_group" {
  name              = "/ecs/${var.service_name}-task"
  retention_in_days = 1
}

Application Load Balancer (ALB)

An ALB will connect our API Gateway to Fargate task containers running in private subnets of our VPC. The Elastic Container Service will use this load balancer to distribute requests across tasks in a cluster, as well as register/drain task containers with the ALB during deployment or scaling. Pass a list of vpc_subnet_ids corresponding to the private subnets in our VPC.

resource "aws_lb" "nodejs" {
  name            = "${var.service_name}-alb"
  internal        = true
  subnets         = var.vpc_subnet_ids
  ip_address_type = "ipv4"
}

resource "aws_lb_target_group" "nodejs" {
  name            = "${var.service_name}-alb-tar"
  target_type     = "ip"
  ip_address_type = "ipv4"
  port            = 80
  protocol        = "HTTP"
  vpc_id          = var.vpc_id
}

resource "aws_lb_listener" "nodejs" {
  load_balancer_arn = aws_lb.nodejs.arn
  protocol          = "HTTP"
  port              = 80

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.nodejs.arn
  }
}

API Gateway v2

An HTTP API Gateway will receive requests from the internet and proxy them to the ALB. aws_apigatewayv2_vpc_link is the key resource here, connecting to the ALB running in our private subnet. integration_type and protocol_type are both "HTTP", and we are forwarding all routes route_key = "ANY /{proxy+}" to the ALB.

resource "aws_apigatewayv2_api" "api" {
  name          = "${var.service_name}-api"
  protocol_type = "HTTP"
}

resource "aws_apigatewayv2_vpc_link" "nodejs" {
  name               = "${var.service_name}-api-link"
  security_group_ids = var.vpc_security_group_ids
  subnet_ids         = var.vpc_subnet_ids
}

resource "aws_apigatewayv2_integration" "nodejs" {
  api_id           = aws_apigatewayv2_api.api.id
  integration_type = "HTTP_PROXY"
  integration_uri  = aws_lb_listener.nodejs.arn

  integration_method = "ANY"
  connection_type    = "VPC_LINK"
  connection_id      = aws_apigatewayv2_vpc_link.nodejs.id
}

resource "aws_apigatewayv2_route" "nodejs" {
  api_id    = aws_apigatewayv2_api.api.id
  route_key = "ANY /{proxy+}"

  target = "integrations/${aws_apigatewayv2_integration.nodejs.id}"
}

resource "aws_apigatewayv2_stage" "default" {
  api_id      = aws_apigatewayv2_api.api.id
  name        = "$default"
  auto_deploy = true
}

Elastic Container Service

We use ECS to run our Docker container, creating a Service, Cluster and Task. The Service will use the ALB referenced above, the Cluster will use Fargate capacity providers, and the Task will have a container definition referencing the repository url and image.

The container definition also loads secrets from Secrets Manager as environment variables for the container runtime, as well as port mappings to the host. We expose port 3000 on the container, and use the same port for the load_balancer configuration on the ECS service.

The Service will manage the ALB, registering and deregistering targets in the ALB's target group for each running Task.

resource "aws_ecs_task_definition" "nodejs_task" {
  depends_on = [
    null_resource.task_ecr_image_builder
  ]

  family                   = "${var.service_name}-task"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = var.task_cpu
  memory                   = var.task_memory
  execution_role_arn       = aws_iam_role.ecs_task_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_execution_role.arn
  container_definitions    = <<TASK_DEFINITION
[
  {
    "name": "${var.service_name}-container",
    "image": "${aws_ecr_repository.task_repository.repository_url}:${data.aws_ecr_image.task_image.image_tag}",
    "cpu": ${var.task_cpu},
    "memory": ${var.task_memory},
    "essential": true,
    "portMappings": [
      {
        "hostPort": 3000,
        "protocol": "tcp",
        "containerPort": 3000
      }
    ],
    "environment": [
      {
        "name": "NODE_ENV",
        "value": "${var.node_env}"
      },
      {
        "name": "FOREST_AGENT_URL",
        "value": "${aws_apigatewayv2_api.api.api_endpoint}"
      }
    ],
    "secrets": [
      {
        "valueFrom": "${aws_secretsmanager_secret.task_secrets.arn}:FOREST_ENV_SECRET::",
        "name": "FOREST_ENV_SECRET"
      },
      {...}
    ],
    "logConfiguration": {...}
  }
]
TASK_DEFINITION

  runtime_platform {
    operating_system_family = "LINUX"
  }
}

resource "aws_ecs_cluster" "nodejs" {
  name = "${var.service_name}-ecs-cluster"

  configuration {
    execute_command_configuration {
      kms_key_id = aws_kms_key.nodejs.key_id
      logging    = "OVERRIDE"

      log_configuration {
        cloud_watch_encryption_enabled = false
        cloud_watch_log_group_name     = aws_cloudwatch_log_group.nodejs.name
      }
    }
  }
}

resource "aws_ecs_cluster_capacity_providers" "nodejs_fargate" {
  cluster_name = aws_ecs_cluster.nodejs.name

  capacity_providers = ["FARGATE"]

  default_capacity_provider_strategy {
    base              = 1
    weight            = 100
    capacity_provider = "FARGATE"
  }
}

resource "aws_ecs_service" "nodejs" {
  depends_on = [
    aws_lb_listener.nodejs
  ]

  name          = "${var.service_name}-ecs-service"
  cluster       = aws_ecs_cluster.nodejs.id
  desired_count = 1

  # Track the latest ACTIVE revision
  task_definition        = aws_ecs_task_definition.nodejs_task.arn
  enable_execute_command = true

  network_configuration {
    subnets = var.vpc_subnet_ids
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.nodejs.arn
    container_name   = "${var.service_name}-container"
    container_port   = 3000
  }
}

And that's about it! Here's another copy of the header image to reference the services managed by Terraform:

Conclusions

Once put together, this is a pretty nice setup to run a Node.js app as serverless! The AWS services work well together, such as the API gateway providing a link into a private VPC, and the ECS service managing a load balancer to register/drain task containers automatically. But they aren't so tightly-coupled to lock you in - it's possible to bring your own load balancer or container runtime, and swap out the corresponding AWS service.

A few things I tried did not work:

• At first attempt, I deployed the Fargate task into a public subnet, in hopes that it would be reachable through a public IP address. This would represent a big shortcut, potentially eliminating the gateway and load balancer. I had trouble getting the task deployed though, as it couldn't load secrets from the secretsmanager. Ultimately, this would be an inferior setup from a security perspective, as it's much better to not expose the container network to the Internet.
• Using a network load balancer (NLB), instead of an application load balancer (ALB), would not see tasks as healthy when ECS with a TCP health-check. My guess is that Express doesn't respond to TCP requests in a way that the NLB recognized as healthy. After switching to an ALB, the health-check is done over HTTP, and I could see the requests hitting my health-check endpoint in Express.

But why?

You've made it to the end of this post! And you might be wondering why we run integration tests like this?

We are using Forest Admin to demonstrate how JumpWire can be plugged into an existing application without modifying any of the application's code or features. It's as simple as providing Sequelize, a popular node.js ORM used by Forest Admin, with a PostgreSQL URL for a JumpWire proxy. No other changes are necessary, we are taking the Forest Admin agent off-the-shelf and running it with JumpWire.

Happy Coding!