AWS Lambda Powertools: Idempotency, A Deeper Dive

In an earlier article, I wrote about Making all your APIs Idempotent. I shared why idempotency is essential and introduced the AWS Lambda Powertools for Python, which provides a utility for adding idempotency to your Python built lambdas in a quick and easy to use way. My article focused on an API as the reason for requiring idempotency and, while accurate, it is not the only use-case for the Powertools Idempotency utility. In this article, I dive deeper into the Powertools utility and highlight more features and use cases.

How Idempotency Works

The idempotency utility is quite a complex feature wrapped up into a simple package to make it easy and approachable to use. The idempotent utility follows these steps:

1. Calculate a unique idempotent token using the request payload and check the idempotent store to see if the transaction has been completed or is currently in flight.
2. If completed, the utility will return the stored result from the first execution; If in-flight, the utility will raise an IdempotencyAlreadyInProgressError exception signalling to the caller it is safe to retry the operation.
3. If no transaction exists, the utility will save an in-flight record into the store and call the function to process the transaction. When the function completes, the utility will keep a copy of the response so that future invocations can receive the same result.

At its core, the idempotent implementation for power tools derives a hash value (md5 by default) for the idempotent token of the transaction. Therefore, calculating this unique token is the most critical part that we need to understand to ensure idempotency works correctly.

Calculating the Idempotent Token

AWS Powertools has your back in this department and will take on board sane, consistent defaults that will work in more straightforward use cases by calculating the idempotent token using the entire message body. However, if your use case is not simple, you must understand how your data changes between invocations. When processing an AWS Lambda event, many of the fields within the event will contain data that changes between each request, for example, "requestId" and "requestTime" in the AWS API gateway Proxy event, which vary for each API invocation.

For non-default hash keys, you will want to use the IdempotencyConfig object and ensure you set the event_key_jmespath string to allow powertools to extract the unique idempotency token from the Lambda event for the hash key.
Understanding how JMESPath and the Powertools JMESPath powertools_json functions decode strings to JSON objects is essential for the idempotency utility.

Making a JSON REST API Idempotent

An excellent example is using the utility with API gateway proxy events for a JSON REST-based API. Unfortunately, the AWS API Gateway Event's body attribute is a string value that is not ideal when trying to assert the idempotency of your JSON Payload.
By definition, attributes in a JSON Payload for a REST API have no order associated with them, meaning the following examples are considered identical in terms of an API transaction:

{
    "transaction_id": "bd784218-31cc-46be-b66a-a9c38b9a2fe5",
    "user_id": "10000233220",
    "email": "me@email.com"
}

is the same as,

{
    "email": "me@email.com",
    "user_id": "10000233220",
    "transaction_id": "bd784218-31cc-46be-b66a-a9c38b9a2fe5"
}

The Powertools idempotency implementation considers this detail, but only if you ensure the body is decoded into a JSON object using the powertools_json built-in JMESPath function. To achieve this outcome for the above example, you would need to set up the IdempotencyConfig as follows:

config = IdempotencyConfig(event_key_jmespath="powertools_json(body)")

With the powertools_json built-in function, the data given to the Idempotent Utility's hash generation function will be transformed into a python Dictionary object, allowing the hash generator to convert the data into a sorted JSON serialized string. In this way, your API will remain idempotent regardless of the attribute ordering in the API JSON body.

When choosing the parts of your message payload for the unique idempotency key, it is critical to realize that a string value can contain whitespace and newlines, affecting the hash outcome.
I strongly recommend reviewing the JMESPath tutorial page to understand how the JMESPath utility works; it is a powerful tool in your python arsenal and is essential in ensuring the Powertools idempotency utility works for you correctly.

Idempotent Decorators for Everything!

The core of the idempotency utility are the python function decorators (yes, there is more than one!). A recent addition to AWS Lambda Powertools for Python is the idempotent_function decorator, which provides idempotency to any synchronous Python function. So now you can add idempotency to absolutely anything, not just Lambda handlers!

Making any function Idempotent

Using the standard idempotent_function decorator, you can tell Powertools how to calculate the idempotency key using any of the function parameters. The IdempotencyConfig is also available for customizing how the utility will work utilizing the event_key_jmespath as mentioned above.

The following code example showcases applying the idempotent_function decorator to the SQS batch processor function making your SQS queue processing completely safe from processing the same message multiple times.

Payload Validation

The AWS Builder's Library article on Making retries safe with Idempotent APIs covers the need for payload validation when data that is not part of the idempotent token changes between requests. You need to consider these scenarios carefully so you do not confuse your API consumers; Powertools has a payload validation feature you can leverage to cover this situation.

Testing Your Solution

Once you have designed and implemented idempotency in your solution you will want to test it thoroughly before making it live. You can test your solution using a local dynamo DB or mocks to save on cloud costs; Powertools covers code testing scenarios completely in their online documentation.

Conclusion

My everyday work is building and training Serverless teams and making sure they build reliably, with resilience and taking note of the AWS Well-Architected Principles documented in the Serverless Lens. I always recommend developing on AWS Lambda with the Python runtime since I can rely on teams meeting the Well-Architected principles through the AWS Lambda Powertools. For groups who insist on Typescript or .NET, I strongly urge them to use Python since it has Powertools, the swiss army knife that AWS Lambda developers need for every project!

If switching your development to Python for AWS Lambda Powertools is not for you, I recommend heading over to the AWS Powertools Roadmap and supporting the Typescript and .NET feature requests by adding a 👍.