Or why the vast difference between Lambda Powertools per language features matter
Written by Samuel Durand
Intro
If you work with AWS Lambdas regularly, then you should already be using Lambda Powertools. This library has a really great set of features and utilities, maintained by AWS, and specifically designed and language-optimized for AWS Lambda. This set includes simple features that improve basic Lambda capabilities, such as structured logging and improved tracing, but also more elaborate features normally requiring a lot of boilerplate (i.e., Idempotency - but more on this later).
Due to their popularity, the Lambda Powertools are often mentioned in Lambda related articles, but one feature can quickly become frustrating. Some of the best, most often mentioned features, such as feature flags management only seem to exist in one or two of the Lambda languages (and not always in a production ready state). Comparing the Lambda Powertools documentation for Python, Java, Typescript, and .Net, quickly reveals that this is not a rare occurrence, and that the number of features available is extremely dependent on which language you work with.
The goal of this article is to show you how much the features available in the Powertools differ per language, and why it might be judicious to take it into account when deciding what language you will use to develop your Lambdas.
What's so great about the Powertools?
The Lambda Powertools are designed to make the developer's life easier by providing:
- less boilerplate for both simple and more complex features (such as logging)
- opinionated code optimized for Lambda
- an easier adoption of AWS best practices
- a softer learning curve for newcomers to Lambda and Serverless in general
- for most languages, a set of Lambda layers maintained by AWS
They support a limited number of languages, which currently are Python (2019), Java (2020), Typescript (mid-2020), and .Net (2023). The team also plans to add support at some point next year for Go, but there are no official plans for Ruby yet.
To demonstrate what the Powertools can do, let's have a look at one of the most popular features they offer: the structured logging feature.
Structured logging
The standard logs for a Lambda are usually not very detailed, not easy to read or to parse, and lack contextual information.
By simply adding a couple Powertools decorators in your code, you can make Lambda logs considerably more useful. The below example shows what the content of each log would look like if you configured the Powertools to add almost every detail available, including the original payload.
{
"cold_start": true,
"function_arn": "arn:aws:lambda:eu-central-1:11111111111:function:PowertoolsExampleStack-S3LambdaPoweredUp6F7EA-BC4pVVUUc5Dd",
"function_memory_size": 128,
"function_name": "PowertoolsExampleStack-S3LambdaPoweredUp6F7EA-BC4pVVUUc5Dd",
"function_request_id": "68e8db49-65f7-430d-9514-dd2af37ce5d3",
"level": "INFO",
"message": "Lambda invocation event",
"service": "s3Uploader",
"timestamp": "2023-07-25T15:44:34.777Z",
"xray_trace_id": "1-64bfede2-43e5e22d5f1b1530bea5ef7",
"event": {
"resource": "/upload-powered-up",
"isBase64Encoded": false,
[...] // Full API Gateway event details
}
}
In this log, some fields can be especially useful:
- cold_start: indicates if this function was subject to a cold start
- function_request_id can be used to figure out if this execution is a retry of a previously processed payload
- xray_trace_id: can be used to trace a particular request just by searching the logs and without paying for X-Ray
- service: this helps identify Lambdas composing a particular Service entity
To configure this, very little change to your code is required. It basically comes down to adding a few decorators.
This is just an example of how powerful Lambda Powertools can be with very little effort invested. But that's all I will show here, since the topic of this article is not to give a tutorial on how to use the Powertools. If you want to see more on how it can be configured in a full CDK project, including an integration of the Powertools' Idempotency feature, have a look at my GitHub repository.
Most common features
Now that you have an idea of the sort of functionality we can achieve with the Lambda Powertools, let's have a look at some of the most common features that most languages support.
Logging (link)
As demonstrated earlier, this feature provides structured JSON logging with a lot of additional contextual data, including the original event payload, if requested.
Tracing (link)
This feature is a thin wrapper for the AWS X-Ray service, which provides tracing capabilities. It generates traces in X-Rays with additional metadata and time datapoints to help debugging, such as the cold start, exception messages, and response bodies. It also provides helpers to add your own tracing points and will be disabled automatically if not running inside the Lambda environment (great for local development).
Custom Metrics (link)
This feature uses the CloudWatch Embedded Metric Format to allow asynchronous recording of custom metrics in CloudWatch. The metrics are recorded in the logs, then extracted by CloudWatch as metrics at a much lower cost than creating custom metrics via CloudWatch APIs.
Parameters (link)
Using this feature, you can automatically retrieve parameters stored in the Parameter Store, Secrets Manager, AppConfig, or in DynamoDB. The parameters will be cached for a configurable amount of time and passed to your handler at runtime. You can even use your own custom parameter store if desired.
Idempotency (link)
Event sources rarely provide exactly-once delivery. This is why most event handlers need to be idempotent, which means they have to make sure that processing the same event multiple times will have the same result, no matter how many times it happens. This can be achieved easily with this feature, which relies on DynamoDB for the storage part of the mechanism.
Feature flags (link)
This utility can be used to configure your code to enable or disable features, either statically (e.g., a feature is not yet ready for production) or dynamically (e.g., per user). Each flag value can be retrieved either from AppConfig or from your own custom configurations store.
Streaming (link)
Lambdas usually have a very limited amount of memory, which can be an issue if you need to process very large S3 objects. You can either increase this value (up to 10Gb) and deal with the associated cost, or you can use the Powertools' streaming utility to process this enormous file a few Mb at a time.
Batch failure processing (link)
When using SQS, Kinesis or DynamoDB streams as an event source with batch processing, if you fail to process one of the events, you have the option to tell those services which event has been successfully handled, and which was not. This process can be extremely simplified by using the Batch Processing feature from the Powertools library.
Validation (link)
This utility provides JSON Schema validation for events and responses, including JMESPath support to unwrap events before validation.
Large SQS messages handling (link)
When using SQS, the size of the messages is limited to 256kb. If you need to send a larger event, one of the solutions is to use the AWS SDK to upload the large payload to S3, then to send the reference to this file to SQS. You will then need to download the payload from S3 in the event handler before using it. The Powertools' utility can be used to automatically retrieve the content of that payload from S3 automatically during the event processing.
Not all languages are equal
All those features are great, but unfortunately, not all features are available for all languages. And even when they are present, they are not always implemented with the same level of support and readiness (prod ready, preview, beta...). This also applies to the features that I previously called "most common." The table below highlights this disparity in language support.
It should be pretty obvious that Python and Java are the languages with the most production ready features. This makes sense since they were the first languages added to the Lambda Powertools. You might have also noticed the last row where I mention that Python has a lot more features, and by that I meant that the Python Lambda Powertools contain at least 7 more unique features than any other language.
This inequality is probably due to the very high difference of activity level between each language repository, which often happens with open source and community supported projects. If you look at the number of PRs, and at the commit dates and frequency on each library's repository on GitHub, it is clear that Python is by far the most lively repository.
Choosing the right language
Selecting the language to use for your Lambda usually comes down to the following questions:
- what language is my team most comfortable with?
- what language offers me the features/libraries that are most adapted to my use case?
- how fast does the Lambda startup/execution need to be?
- are cold starts a concern?
Those are the usual questions, but considering how useful and time saving some of the features of the Lambda Powertools are, it can be a good idea to also include it into the decision process.
Imagine yourself in the situation where:
- you are building a very simple application processing SQS events referring to large objects stored in S3
- each large object should be downloaded from S3 and split into parts sent to EventBridge as an event
- the logic is simple but you must make sure that events are not processed twice
- you want to stream the S3 object content instead of downloading it all in one go (to avoid using a lot of memory)
- your team is very skilled at Typescript, but has also some experience in Python
In this situation it might make sense to choose Python, despite your team having less experience with it. The code will be be relatively simple anyway, but would be even simpler and shorter if you use the idempotency and S3 streaming features from the Python Powertools library.
Conclusion
Lambda Powertools are a very powerful set of utilities which most Lambdas should be using. But you should never assume that a feature is present in a specific language just because you heard about it, or that this feature is production ready. Furthermore, considering the value of this toolset and depending on your particular use case, it might be worth considering the Powertools available features per language when choosing a programming language for your Lambda. You might save yourself a lot of pain and time by switching to a language you are less familiar with, but for which the capabilities that you need are already implemented, tested and optimized.
Keep in mind though that, just like any external dependency, adding the Lambda Powertools to your project will add some constraints on your project. And it might incite you to use a lot of fancy features you might not really need, and that might impact the overall performances of your application. Furthermore, those features are implemented in an opinionated fashion, which might not fit with your way of working, and offer a more or less limited number of configuration options, depending on the feature and language.
Note that since the AWS Lambda Powertools team recently became an official AWS Team, it should provide more official support and help to fix this features disparity in the medium to long term.
I hope this advice will help you in your future projects, and as usual, if you have questions, or you would like to hear about my next articles, then follow me on Linkedin or contact me by email at samuel.durand@kreuzwerker.de.
Top comments (0)