Improving CDK development cycle

Here at Instil we have been working on a fully serverless car insurance platform that utilises the AWS CDK for deployments. One of the lessons we learnt early on in this project is that there is no substitution for deploying and testing our code in AWS; each of our developers have their own AWS accounts that they regularly deploy code to. However with CDK sometimes these deployments can take a while...

As an example, we have some versioned Lambda functions that run pre-traffic test functions with CodeDeploy to verify that our deployed code is configured and behaving correctly. Typically these can take 3-4 mins to deploy, run tests, and switch traffic when using the cdk deploy [stackname] command.

While this functionality is extremely useful, those 3-4 mins spent waiting on deployments add up, thankfully CDK has a few ways of reducing this feedback loop that we should all be aware of.

The AWS CDK Toolkit (aka the CDK CLI) page on GitHub covers all of the CLI commands in detail and is worth keeping up-to date with for any future improvements!

What is hotswapping?

You can pass the --hotswap flag to the deploy command:

$ cdk deploy --hotswap [StackNames]

If you have only changed the code of a Lambda function, and nothing else in CDK this will attempt a faster deployment by skipping CloudFormation, and updating the affected resources directly. If something cannot be hot swapped then CDK will fall back and perform a normal deployment.

Currently hotswapping is supported in the following scenarios:

• Code asset (including Docker image and inline code) and tag changes of AWS Lambda functions.
• AWS Lambda Versions and Aliases changes.
• Definition changes of AWS Step Functions State Machines.
• Container asset changes of AWS ECS Services.
• Website asset changes of AWS S3 Bucket Deployments.
• Source and Environment changes of AWS CodeBuild Projects.
• VTL mapping template changes for AppSync Resolvers and Functions.

This list could update so be sure to check out this page for the most up-to date list.

The hotswap command deliberately introduces drift in CloudFormation stacks in order to speed up deployments. For this reason, only use it for development purposes. Never use this flag for your production deployments!

How does CDK know when to hotswap?

Behind the scenes CDK uses another useful CLI command: cdk diff.

The diff command computes differences between the infrastructure specified in the current state of the CDK app and the currently deployed application.

If you run cdk diff after making only code changes you will see output that looks like this:

Stack MyStack
Resources
[~] AWS::Lambda::Function
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac.zip
 │       └─ [+] 21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0.zip
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac
         └─ [+] asset.21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0

Notice the changes only refer to an Asset and nothing else in the stack. This is considered “hot-swappable” and will be hot swapped successfully by using the --hotswap flag.

Running cdk diff when there are non-code changes would output something like this:

Resources
[~] AWS::Lambda::Function
 ├─ [~] Code
 │   └─ [~] .S3Key:
 │       ├─ [-] 113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac.zip
 │       └─ [+] 21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0.zip
 ├─ [~] Environment
 │   └─ [~] .Variables:
 │       └─ [+] Added: .MATTHEW_WAS_HERE
 └─ [~] Metadata
     └─ [~] .aws:asset:path:
         ├─ [-] asset.113f08cf527f79c4a1b851dce481b9567a4e7c6dd5a9e0f47b35692837df05ac
         └─ [+] asset.21127a377c84d613789121f2be5ee85c6e285d40d6fdf0db50467ec44998faa0

Notice the changes includes an environment variable addition as well as code changes. This change could not be hot swapped and would fall back to a normal deployment.

Watching for changes

CDK also provides a nice command called cdk watch, this uses the hotswap functionality and continuously monitors the files of the project, and triggers a deployment whenever it detects any changes.

https://github.com/aws/aws-cdk/tree/master/packages/aws-cdk#cdk-watch

A nice feature of this command is that it will also tail the CloudWatch logs after deploying your code! Hopefully reducing your feedback loop even more.

Summary

In our testing using --hotswap for our versioned lambdas shaved ~4 mins off our deployment times!

To ensure that hotswaps function correctly you want to adhere to the CDK principle of keeping your stacks deterministic, for example avoid dynamically generating properties of your constructs (e.g. using a timestamp, or network lookups).

If you or your team want to learn more about CDK, be sure to check out our TypeScript for AWS Serverless Course.