Chaos engineering is the practice of performing controlled experiments on our systems in order to learn new things about how it behaves and to build confidence both in the system and in our organization. It helps us build reliable and robust distributed systems.
Creating modern applications using serverless technology and managed services means that anyone can build distributed and highly available systems without worrying about the underlying infrastructure. But it’s not all fun and games. This talk I did at AWS re:Invent 2019 with the title Performing chaos engineering in a serverless world explains some of the challenges with serverless, common weaknesses in serverless applications as well as challenges with using chaos engineering in serverless:
View the slides for the talk
Inject failure in our functions
AWS Serverless Hero Yan Cui has written several articles about latency injection for AWS Lambda, see How can we apply the principles of chaos engineering to AWS Lambda? and Applying principles of chaos engineering to AWS Lambda with latency injection from October 2017. These articles explain why we could and should use chaos engineering in our serverless applications as well as showing examples on how to do it.
AWS Principal Developer Advocate Architecture Adrian Hornsby expanded on this by first creating a Lambda layer and later a Python library for failure injection, chaos_lambda, giving developers an easier way to get started with chaos experiments for AWS Lambda. Just install the library, wrap your functions with the appropriate failure mode and you are ready to start injecting failure!
To reach the same level of simplicity for NodeJS developers I late last year created an NPM package called failure-lambda. The goal with failure-lambda is in short to have an easy way to do failure injection in AWS Lambda using several different failure modes. To make it even easier I decided to use a single wrapper and instead have the failure mode selectable. That way you don’t have to make code changes if you want to switch between for example latency or exception injection, you just change a setting.
As we know serverless isn’t just about AWS and chaos engineering for serverless isn’t only about AWS Lambda. For that reason, there are now also the same failure injection options for NodeJS developers building serverless using Azure Functions and Cloud Functions. This with the NPM packages failure-azurefunctions and failure-cloudfunctions.
Failure modes and rate of failure
Even though this all started with latency injection as in Yan Cui’s articles, latency is far from the only possible failure we can have in our serverless applications. In failure-lambda, failure-azurefunctions and failure-cloudfunctions there are now five different failure modes to choose from:
- Latency
Injects latency to the executed function, controlled using a minimum and maximum span of milliseconds. This can for example be used to simulate service latency or to test and help set your timeout values.
- Exception
Throws an exception in the function. Helps you test how your application and code handles exceptions.
- Status code
Your function will return a status code of choice, for instance 502 or 404 instead of the normal 200. This gives you the possibility to test what happens when there are errors.
- Disk space
Will fill your temporary disk with files to create a failure. If you’re using disk to store temporary files you can test how your application behaves if that disk gets full or you are unable to store to it.
- Denyhlist (courtesy of Jason Barto)
Blocks connections to specified hosts. Use to simulate services or third parties being unavailable.
All these failure modes can be used together with a rate of failure that you set. The default is to inject failure on every invocation but in reality, it is likely that for example a third party is unavailable on 50% of the calls made to that host or that an exception is thrown on a quarter of the invocations. Setting rate will allow you to achieve this.
Getting started
All three NPM packages contain step by step instructions on how to get started using them and injecting failure. There are also example applications that you can use to try it out.
https://github.com/gunnargrosch/failure-lambda#how-to-install
https://github.com/gunnargrosch/failure-azurefunctions#how-to-install
https://github.com/gunnargrosch/failure-cloudfunctions#how-to-install
If you want an even more in-depth explanation, I will show you how to install and use them all in the next article in this series.
Top comments (2)
This is a great post thanks Gunnar!
Thank you, Alex! 🙏