Strategies for a Faster Lambda Development

Starting with AWS Lambda can be challenging, especially when optimizing and streamlining the development process. While running functions directly in the AWS Console is the simplest and quickest approach, it becomes tedious when adding packages, as it requires zipping and uploading them to the console – a method that is rarely used in practice.

An alternative workflow involves setting up Infrastructure as Code (IaC) tooling, such as AWS CDK and running deployment commands like sam deploy with AWS SAM. These tools handle code uploading and simplify the process.

Although using IaC tooling is a step forward, it can still be slow, as it often requires updating CloudFormation, which can take time. In this blog post, we will explore various strategies and tools to make Lambda development faster and more efficient.

Strategy: Bypassing Cloudformation

Many tools which are on top of Cloudformation (CFN) like CDK or AWS SAM, synthesize their code to a CFN- template before deploying to the cloud. That means they create a CFN template and make use of AWS native IaC-service. It is nothing wrong with that, it is just freaking slow.
Let's have a look at the cdk deploy command

To begin, the command creates a CFN template and uploads it, along with a ZIP file, to the assets-bucket (which is created when cdk bootstrap is initiated, more information can be found here). The Cloudformation stack is then either created or updated, with drift detection being conducted to ensure any manual changes are accounted for. Finally, an AWS Lambda function is created alongside the stack to retrieve the ZIP-file.

We can automate many steps with this command, but updating Cloudformation for small changes takes a lot of time and can disrupt the development process. Let's explore cdk deploy --hotswap --watch, also known as cdk watch. This command uses --watch to detect any file changes and trigger updates automatically.

By using this command, we skip the CloudFormation step and directly call the Lambda-API. Additionally, the development experience is improved by running a watcher that detects file changes, giving the feeling of a hot module reload. However, it's important to note that this process may cause discrepancies between your CloudFormation template and the deployed application, and not all resource changes can be 'hot-swapped' (Ref. here). Therefore, it's recommended to only use this in development and use the cdk deploy command in production.

AWS SAM has also introduced a similar concept called SAM accelerate.

SST

The first time when I saw a 'hot swap' was at SST.

SST is a framework that makes it easy to build modern full-stack applications on AWS.

SST sits on top of CDK, and they had first the idea of bypassing Cloudformation. They call it Live Lambda Development.

Live Lambda Development or Live Lambda is feature of SST that allows you to debug and test your Lambda functions locally, while being invoked remotely by resources in AWS. It works by proxying requests from your AWS account to your local machine.

I am a big fan of SST because they do a much better job than plain CDK ❤️

• Much better documentation, (why the heck I cannot search in the CDK API doc?)
• Much better development experience (DX)
• Faster Development and Deployment
• Local UI locally for triggering the Lambda

Generally, if you consider developing Serverless Apps, you should consider SST.
You won't be disappointed ;)

Strategy: Run Lambda locally

Another strategy could be to run an emulator locally which simulates your cloud provider.
An emulator is a service with a bunch of APIs running locally in your container or as a web service.

Serverless Framework Offline

The Serverless Offline-plugin is the only local emulator which does not require Docker but creates a local HTTP Server in NodeJS (see the diagram above). It simulates an API gateway and simply invokes the lambda locally.

This only works with Lambda which is invoked via an API Gateway. This great but limited to API Gateway.

Nevertheless, there are a few plugins for Eventbridge, S3 Events, and DynamoDB local for local development which are sitting on top of Serverless Offline.

This is easy to set up and really fast and highly recommended when using Serverless Framework.

Localstack

Localstack is a cloud service emulator that runs in a single container on your laptop or in your CI environment.

It can be installed and run and make sure Docker is installed and also running

pip install localstack

# or in with docker
# docker run \
#   --rm -it \
#   -p 4566:4566 \
#   -p 4510-4559:4510-4559 \
#   localstack/localstack

localstack start -d # in Daemon mode (running in the background)

It offers a bunch of ports to run against it. But most of the core services are running on port 4566.
You could now do all the AWS-CLI- commands against this endpoint by appending the flag --endpoint-url=http://localhost:4566. Or you can use a tool called awslocal.
Localstack also comes with a lot of integrations for your development tools such as CDK, Terraform, AWS SAM and many more.

Let's have a look at cdklocal. I would recommend installing it not globally but in your project.
pnpm install aws-cdk-local. Once you have created your CDK-stack, you could run npx cdklocal deploy. But whoops...

✨  Synthesis time: 4.89s

LambdaStack: building assets...


 ❌ Building assets failed: Error: Building Assets Failed: Error: LambdaStack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
    at buildAllStackAssets (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:383:115268)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CdkToolkit.deploy (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:383:143485)
    at async exec4 (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:438:51799)

Building Assets Failed: Error: LambdaStack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)

Okay, you need to run npx cdklocal bootstrap first and then npx cdklocal deploy.
The CLI shows you exactly the same as the CDK-CLI which is pretty cool.
After deployment, you see in the Terminal something like

LambdaStack: creating CloudFormation changeset...

 ✅  LambdaStack

✨  Deployment time: 25.17s

Outputs:
LambdaStack.DemoRestApiEndpointF0DDDDE8 = https://i6d181wlsa.execute-api.localhost.localstack.cloud:4566/dev/
Stack ARN:
arn:aws:cloudformation:eu-central-1:000000000000:stack/LambdaStack/90063dd4

✨  Total time: 27.94s

If you have implemented an API Gateway like me, you will even see an endpoint as an Outputs.
cdklocalcan even execute npx cdklocal watch and do hot-swap.

However, you may notice that it involved too many steps, and it's faster to use hot-swap directly with CDK.

AWS SAM local

The all-rounder AWS SAM can also run an emulator for its SAM template. It offers sam local with many subcommands:

• sam local generate-event – Generate AWS service events for local testing.
• sam local invoke – Initiate a one-time invocation of an AWS Lambda function locally.
• sam local start-api – Run your Lambda functions using a local HTTP server.
• sam local start-lambda – Run your Lambda functions using a local HTTP server for use with the AWS CLI or SDKs.

Strategy: Test-Driven Development (TDD)

Test-Driven Development (TDD) is an effective approach for breaking down Lambda code into smaller, single functions. This helps to structure the code better and gives the developer more confidence in the implementation. By breaking down your code into smaller, individual functions, you can easily write unit tests. Your Lambda handler should be the culmination of all your functions.

When using TDD, you can use a test runner such as Jest or Vitest for NodeJS, or pytest with pytest-watch for Python. This will help to ensure that the code is being tested for correctness and that any errors or bugs are quickly identified and addressed. Additionally, the test runner can be used to automate the testing process, saving time and effort.

When using TDD and Lambda, mocking AWS services can be challenging. Your Lambda's behavior may not match expectations, making it difficult to identify all edge cases. Despite this, the advantages of TDD are worth it, as it ensures the code works correctly and detects and resolves errors or bugs before they become a problem in production. This increases confidence after deployment.

Conclusion

We have seen some strategies for developing Lambda faster by leveraging tools that are bypassing Cloudformation such as SAM Accelerate or CDK's hotswap. It is also okay to gain confidence by using local emulators as they can be integrated into your development workflow.
Let's summarize

According to Yan Cui's blog post titled 'My testing strategy for serverless applications', using an emulator like localstack is a waste of time and breaks in a mysterious way. However, using a combination of different testing strategies can increase confidence before deployment. I have personally found that using TDD and hot-swap is effective for faster integration testing.

Let me know, what is your development setup and what tips and tricks you have.