⚡️Combining Serverless Framework & AWS CDK

Serverless Framework is a fantastic tool for defining and managing your Lambda functions. But let's face it; it's not very good at handling other types of resources and infrastructure. CDK is though. This post will take a look at how we can combine the two!

Serverless has excellent utility for describing and managing Lambda functions, but as soon as you need basically any other type of infrastructure, you'll have to resort to raw CloudFormation. Using the CDK with its declarative and straightforward constructs to define infrastructure is more expressive and easy to understand once the infrastructure stack grows. What if we could use them together and utilize each of them for its respective strengths, and what it does best? Unfortunately, there's no native way to use the two tools together, like there is for SAM.

Let's look at how we can work around that and whether or not it makes sense!

TL;DR - give me the code!

🏗 Setting up the project

Let's start by initializing a Typescript Serverless app. To keep things simple, we'll use a Yeoman generator to get us started with a project. Install it with:

npm install -g yo generator-sls-node

and then, in an empty folder, run the generator with:

yo sls-node

Since we'll be using CDK to manage our infrastructure, we'll want to manage CDK, its dependencies, and transpiling the infrastructure code separately from our actual application. Therefore, let's create a folder called infrastructure in the root of our project.

mkdir infrastructure

Next, navigate to the created folder and initialize a CDK project:

cd infrastructure
npx cdk init --language typescript

You should now have a project with the following structure:

- infrastructure # Our CDK infrastructure
    ↳ bin     # CDK executable
    ↳ lib     # infrastructure stack definition
    ↳ test    # infrastructure tests
- src                    # application code
- tests                  # application tests
- serverless.yml # Serverless config

🔨 Building our app

The app we're building will contain a Lambda function, exposed in a HTTP API, which will put an item on an SQS queue for another Lambda to finally process the item.

Let's start by adding an SQS queue to our CDK stack. To do that, we need to first add the @aws-cdk/aws-sqs package:

yarn add @aws-cdk/aws-sqs

I prefer Yarn over NPM, which is the default of the CDK init boilerplate. If you prefer to use Yarn too, remove the package-lock.json so you don't get conflicting lock files.

Then add a queue to infrastructure/lib/infrastructure-stack.ts

// infrastructure/lib/infrastructure-stack.ts
import * as cdk from '@aws-cdk/core';
import { Queue } from '@aws-cdk/aws-sqs';

export class InfrastructureStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const sqs = new Queue(this, 'SQS', {
      queueName: 'sls-cdk-demo-queue'
    });
  }
}

As discussed earlier, we want to use Serverless and CDK for what they do best respectively. That means our API and Lambda functions will be defined in our serverless.yml, seeing as Serverless handles those very capably and brings lots of functionality to interact with them once deployed. They're also the resources most likely to change often, meaning they don't really belong in our more long-living infrastructure stack.

For this simple demo use case, all we need to do to set up our API Gateway, Lambda function and to map the function to the /todo path, is to add the following to our serverless.yml, using the events mapping:

# serverless.yml
...
functions:
  createTodo:
    handler: src/createTodo.handler
    events:
      - http:
          path: /todos
          method: post

For the createTodo function to send anything to our SQS buffer, it's going to need the URL of the queue. It's also going to need the queue's ARN to set up the necessary sqs:SendMessage IAM permission. However, since the SQS queue is not defined in the same CloudFormation stack as our Lambda function, we'll need to export those two properties from our CDK stack so that we can reference them in our Serverless stack.

Cross Stack References

CloudFormation stacks can export a set of properties, as "Outputs" which can then be referenced in other CloudFormation stacks. Conveniently, Serverless has built-in support to reference Outputs of other stacks, which means we can use this to easily transfer information from our CDK stack to our Serverless stack.

In the CDK stack, we start by creating our output variables:

// infrastructure/lib/infrastructure-stack.ts
...
new cdk.CfnOutput(this, 'queueUrl', {
  value: sqs.queueUrl
});

new cdk.CfnOutput(this, 'queueArn', {
  value: sqs.queueArn
});

Now, back in our serverless.yml, we can reference those two variables with the ${cf:stackName.variableName} syntax:

# serverless.yml
...
provider:
  name: aws
  region: eu-north-1
  runtime: nodejs14.x
  iam: # Set up IAM permission for sending messages to the queue
    role:
      statements:
        - Effect: Allow
          Action: sqs:SendMessage
          Resource: ${cf:InfrastructureStack.queueArn}

functions:
  createTodo:
    handler: src/createTodo.handler
    events:
      - http:
          path: /todos
          method: post
    environment: # Set environment variable with the queue URL
      QUEUE_URL: ${cf:InfrastructureStack.queueUrl}

and while we're at it, lets also add the definition for our second Lambda, the queue processor, again by using the cross stack reference to the queue ARN:

# serverless.yml
...
functions:
    createTodo:
    ...
    queueProcessor:
        handler: src/queueProcessor.handler
        events:
      - sqs:
          arn: ${cf:InfrastructureStack.queueArn}

That should be all we need for the infrastructure, all that's left is writing the actual Lambda code.

Lambda Code

Our Lambda code is going to be very simple. The createTodo handler, the one exposed in the API, will take the body of the request and put it directly on the SQS queue. The queueProcessor will then automatically consume the queue. Starting with the createTodo function, add the following:

// src/createTodo.js
import { SQS } from 'aws-sdk';

export const handler = async (event) => {
  const sqs = new SQS();

  const payload = event.body || '';

  await sqs
    .sendMessage({
      QueueUrl: process.env.QUEUE_URL,
      MessageBody: payload,
    })
    .promise();
};

and then let's just pretend to do some processing of the item in the queueProcessor.js:

// src/queueProcessor.js
export const handler = (event) => {
  console.log('Doing some heavy processing right now. I promise! Don\'t come in!', event);
};

That's it, now all we need is to be able to deploy our app along with the infrastructure!

🚀 Deploying

Since our application will live in two separate CloudFormation stacks, we need to do two separate deployments. The Serverless stack will depend on our infrastructure stack, but never the other way around - meaning we should always deploy the infrastructure stack first.

To be able to deploy our app in one step, rather than manually first having to navigate to the infrastructure folder, deploy it, navigate back to the project root and then deploy our Serverless app, we can add a script to our root package.json that does us for us in one swift motion:

// package.json
...
"scripts": {
  "deploy": "cd infrastructure && yarn cdk deploy && cd .. && yarn sls deploy",
...

We can also add a CDK wrapper script that lets us easily run any CDK command without having to navigate to the infrastructure folder.

// package.json
...
"scripts": {
    "cdk": "cd infrastructure && yarn cdk",
...

The cdk script utilizes the fact that Yarn forwards arguments, and thus we can run, for example, yarn cdk diff from the root of our app, even though CDK isn't actually in the root. Neat! 👵

What's even neater, though, is that we can now just run yarn deploy. Since it's the first time we're deploying the app, it'll take a little longer than on subsequent deploys but after about half the time it takes refill your coffee, the app will be up and running! We can verify that it works by sending a POST request to the API Gateway endpoint printed in the output, and by then looking at the logs of the final queueProcessor lambda:

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://abcdefghij.execute-api.eu-north-1.amazonaws.com/dev/todos

yarn serverless logs --function queueProcessor

Conclusions

CDK is excellent at defining and making sense of complex infrastructure and resources. However it does come with quite a bit of overhead. The CDK project itself has dependencies that you'll need to manage, and, most importantly; the infrastructure code is, well, code, which you'll need to maintain, lint and test. On the other hand, being able to test it, is definitely also one of the strengths. It's also generally a good idea to separate your more long-living infrastructure into a separate CloudFormation stack. It minimizes the risk of accidentally modifying, or even deleting, important stateful resources such as a DynamoDB table.

There are ongoing projects that tries to bring CDK constructs natively right into Serverless via plugins, one example being Lift, and other frameworks that are built on top of the CDK, such as Serverless Stack. Lift looks promising, but it doesn't yet support using custom constructs, and many organizations and projects have invested a lot into Serverless Framework and aren't ready to jump ship to an alternative like Serverless Stack.

For smaller projects, or projects where the infrastructure and/or the interaction between the infrastructure isn't complex, using CDK and Serverless together is unlikely to be worth the overhead. For larger, more complex apps, though, where you might want to unit test your resources or just be able to describe them better than with CloudFormation, which quickly gets very messy - it might be worth considering!

If you enjoyed this post and want to see more, follow me on Twitter at @TastefulElk where I frequently write about serverless tech, AWS, and developer productivity!