How to make an AWS Lambda custom layer for python

AWS Lambda isn't truly "serverless", your code must run somewhere, right? Don't answer that, it's a rhetorical question! 😅
Yes! AWS lambda runs on servers!


But the word "serverless" means "I don't really care where it runs as long as it runs! And the actual servers aren't my concern therefore I won't have access to them".
Pay close attention to the last part as sometimes you would need some packages/libraries depending on your language of choice to use in your lambda function (actually most of the time 😂).
For me I use Python, and there is no way to "pip install ..." into the lambda function the packages I need as I don't have access to the servers that runs the function. See the issue here?! 😕

Enter: AWS Lambda Custom Layers

Custom layers are AWS' solution to that problem ☝️. With custom layers you can include the extra "Umph" your amazing Lambda function needs to do its magic. You can read all about them form the AWS Docs.
But I'm not here to direct you to the docs. I wanted to write this to all the busy cloud enthusiasts out there that are too short on time to read the docs and share my experience creating a python custom layer in a short and clear set of steps. (Actually, you must read the related docs always! Even if you are remotely interested in developing cloud workloads 🤷‍♂️)

(I'm on a windows machine and I use Ubuntu WSL for this. AWS recommends the use of Cloud9 to do that as your custom layer must be compatible with Amazon Linux to function correctly on your lambda. But I find WSL satisfactory in MOST cases)

For the demo, I'll just be following the example listed in the docs as it's very clear but I'll remove the clutter and share my own experience 😉.

1- I'll make one directory to group everything I need for this task as follows:

my-lambda
|_ venv-python3.9
|_ python
|_ lambda-layer.zip

This is how it WILL look like but first I'll create my root directory and "cd" into it.

mkdir my-lambda
cd my-lambda

2- You must use the same python version in your local environment as in your lambda function. My Lambda uses Python 3.9 and I'll use that too. You can use whatever Python environment management solution that makes sense for you. I use Conda but pyenv is good.
This is how you do it using Conda.

conda create -p ./venv-python3.9 python=3.9 -y

This will install python in venv-python3.9 directory in my-lambda. After the installation, you must activate this environment as follows:

conda activate ./venv-python3.9

3- I'll be implementing this functions that uses requests package and that's what I'm making the custom layer for

import requests
def lambda_handler(event, context):   
    response = requests.get("https://www.example.com/")
    print(response.text)
    return response.text

This function just requests the "https://www.example.com" website and prints the response. To see what we would expect, use the following command:

curl https://www.example.com/ 

And we should get something like that.

4- I'll pip install the requests package but I'll change its download path using the --target param so that I don't download it in the default site-packages directory and just download it in the my-lambda directory

pip install --target=./python requests

After executing this command, you will see the packages directory containing all the dependencies for the requests package to work.

5- I'll zip the python directory and that's what I'll use as my custom layer. AWS Lambda extracts whatever is in the zip file in the /opt directory and the lambda python runtime automatically supports /opt/python in the PATH env var.

zip -r lambda-layer.zip python

This will produce the lambda-layer.zip file that I'll upload.

6- Now for the AWS stuff, in order to upload your custom layer, we will do the following:
1- Navigate to Layers form the left navigation bar then press create layer

2- Fill the Create Layer form and upload the zip file we have created.
3- Then press create.

4- After your zip file is uploaded and if everything goes well, you will get the following message.

5- Navigate again to your lambda function, scroll down and click Add a lawer.

6- Select a Custom Layer.
7- Choose your recently uploaded layer.
8- Select your version (generally it will be version 1 unless you have previously made changes to it)
9- Press Add

10- To test the new layer, press Test.

11- Configure your test event, the default values will be fine.

12- Press Test again and this time you will see similar results as the curl command

[Optional] Clean up our local environment

In order to clean up, we will have to do the follwing:
1- Deactivate the python3.9 env

conda deactivate

2- Remove the environement

conda env remove -p ./venv-python3.9

3- Delete the my-lambda directory

cd ..
rm -fr my-lambda

And that's it! Now you know how to create custom AWS Lambda Layers 😉.