Struggling to properly package your Python dependencies into your functions? In this post I’ll take a look at serverless-layers
, a Serverless Framework plugin that can do it for you!
Sample project
Given a simple project that has the following directory structure:
serverless-layers-example/
├── requirements.txt
├── serverless.yml
└── src
├── __init__.py
└── functions
├── __init__.py
└── example_function
├── __init__.py
└── handler.py
Where serverless.yml
contains the following:
service: serverless-layers-example
provider:
name: aws
runtime: python3.8
functions:
example_function:
handler: src/functions/example_function/handler.handle_event
The example_function
is a simple function that calls an external API (cat-facts 🐱) using the requests
library and returns the response in the body.
import json
import requests
FACTS_URL = "https://cat-fact.herokuapp.com/facts"
def get_fact():
r = requests.get(FACTS_URL, timeout=5)
return r.json()[0]["text"]
def handle_event(event, context):
fact = get_fact()
body = {
"message": fact
}
response = {
"statusCode": 200,
"body": json.dumps(body)
}
return response
The requirements.txt
file contains the following (generated by running pip freeze > requirements.txt
after running pip install requests
):
certifi==2020.12.5
chardet==4.0.0
idna==2.10
requests==2.25.1
urllib3==1.26.2
After deploying the function with sls deploy
and invoking the function with sls invoke -f example_function
, you’ll be greeted with the following error:
{
"errorMessage": "Unable to import module 'src/functions/example_function/handler': No module named 'requests'",
"errorType": "Runtime.ImportModuleError"
}
Let’s see how serverless-layers
can help us solve the above.
serverless-layers
One of the first hits when searching for solutions to manage Python dependencies in Serverless projects is the popular serverless-python-requirements plugin. However serverless-layers is a plugin that can work just as well.
As the name suggests, serverless-layers
utilizes Lambda layers to manage functions dependencies. It automatically attaches a layer to each function defined in your serverless.yaml
containing the dependencies.
Let’s give it a try!
Install
- Create a
package.json
by runningnpm init
(If not already exists) - Install the plugin
npm i -D serverless-layers
- Add the plugin to the
plugins
section inserverless.yml
- Because we haven’t specified the
deploymentBucket
in theprovider
section, we have to configure thelayersDeploymentBucket
in the plugin configuration in thecustom
section and make sure the bucket exists.
service: serverless-layers-example
provider:
name: aws
runtime: python3.8
plugins:
- serverless-layers
custom:
serverless-layers:
layersDeploymentBucket: <bucketName>
functions:
example_function:
handler: src/functions/example_function/handler.handle_event
Redeploy the function
Invoke the function again with sls invoke -f example_function
. You should now see the response as expected.
What happened
The plugin first checks if the layer for this project and stage already exists by checking if a
requirements.txt
exists at<bucketName>/serverless/<projectName>/<stage>/layers/requirements.txt
.If not, it downloads the dependencies as specified in
requirements.txt
to.serverless/layers/python/
This directory is zipped and uploaded to the bucket. The resulting zipfile can be found the
.serverless
directory and with the example config as above has the name<projectName>-<stage>-<runtime>-default.zip
. You could inspect the contents of this zipfile by runningzipinfo <zipfileName>
.The
requirements.txt
that the layer is based on is uploaded to the bucket
The layer gets attached to the function. This is visible in the
cloudformation-template-update-stack.json
file in the.serverless
directory.
"ExampleUnderscorefunctionLambdaFunction": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Handler": "src/functions/example_function/handler.handle_event",
"Runtime": "python3.8",
"FunctionName": "serverless-layers-example-dev-example_function",
"Layers": [
"arn:aws:lambda:us-east-1:************:layer:serverless-layers-example-dev-python-default:1"
]
},
}
Adding/updating dependencies
When adding or updating the dependencies the same process as described above gets triggered, but only if the requirements.txt
has actually changed, compared to the one stored in the bucket next to the active layer.
Example of the output when changes are detected:
Example of the output when no changes are detected:
Caveats
- At the time of writing the plugin does only perform a simple check to see if dependencies changed. Changing the order of the listed dependencies already triggers an update of the Lambda layer.
Conclusion
serverless-layers
is a handy plugin that can help you with managing Python dependencies using Lambda layers. It takes away the operational overhead of managing Lambda layers and because the dependencies are no longer packaged with the Lambda function itself it drastically reduces its size and time it takes to deploy new versions of your code.
It might even improve cold-start times (check out This is all you need to know about Lambda cold starts#Where the dependency is loaded from matters), but I have not tested that.
serverless-layers
currently only supports dependencies defined through requirements.txt
. There is no support for other Python dependency management tools such as Pipenv/Pipfile or Poetry.
Top comments (0)