Create Lambda Layers in AWS Lambda

For a while, I struggled when it comes to make layers for my functions. I used to download them locally, zip them, upload into S3 when their sizes are big and then create a version. This process takes a long time and the chances you make a defected layer is high when you're using Windows or Mac because of some layers that compile binaries when you download them.

I wrote an article back in 2019 on how to do that with the help of Docker.

How to compile resources for AWS Lambda

Since then, I still gets reads for it. Which triggers the need to create a new simple and extremely fast way to use this amazing feature, Lambda Layers.

I will walk you through what it does and how you can have it in your account. Please note that this process is only (currently) for Python3.8.

What is does?

This script consists of 3 main steps: Create a new layer, Update existing one and read what's inside your latest layer version.

Create a new layer:

Because of the struggle I mentioned at first, this process is time consuming when it comes to the manual, traditional way. So, with providing some key information, the script will create the directory structure, install the libraries with PIP, calculate the directly size to prevent exceeding layer limit, zip it, upload it into newly created S3 bucket (or existing if you have one), and finally, publishing the new layer.

these steps are the minimal that you can do to create a new layer. Please note that some values are hardcoded, it can be easily made dynamic but its out of the scope (currently).

Update existing layer:

Because managing an existing layer could be hard, as you will need to do many steps to maintain the existing libraries and add new ones. This script will get the referenced zip file of the latest version of your layer, download it, add to it, upload it into S3 again, and publish a new layer version. 

Read Layer content:

Because of the problem that log4j made in the couple of weeks ago, and what threat can existing resources made if they were got affected, you will need to monitor your resources and update them accordingly. This action will get the latest layer version zip file, extract it, use PIP to check what's inside and which version you have. This could also help maintaining supported libraries versions.

Enough talking, lets dive into it:

Preparation and execution:

we will go though the steps in order:

S3 steps(if you don't have one already):

1- Login into your AWS account and go to S3.

2- Create a new S3 bucket, keep it in the same region you work in.

3- Set it up as you wish, no red lines are here.

Lambda steps:

1- Go to lambda console and create a new function.

2- Open the function -> Click on Configuration -> Click on Permissions -> click on Role Name.

3- Click on Policy Name -> Edit policy.

4- Paste this policy (edit resource as you wish) -> save it.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "lambda:ListFunctions",
                "lambda:ListLayerVersions",
                "lambda:ListLayers"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "lambda:GetLayerVersion",
                "lambda:DeleteLayerVersion",
                "lambda:AddLayerVersionPermission"
            ],
            "*"
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "logs:CreateLogStream",
                "lambda:PublishLayerVersion",
                "s3:ListBucket",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

5- Go to GitHub and get the code -> copy the whole code in code.py and paste it into your function.

GitHub Repo

6- Click on Test and paste this json (all fields are required)

{
  "layer_name": "LAYER NAME",
  "s3_bucket": "YOUR S3 BUCKET",
  "libraries": ["LIBRARIES"],
  "action": "create_new"
}

NOTE: in action key, there are 3 valid values: create_new, update, read_only

7- Test the function is what triggers it, give it a try.

These are a proof on this script executions:

Feel free to adjust the code as you wish, and let me know if you have any issues.

Conclusion:

This is a way of doing this procedure, I'm pretty sure you have your own way too. let me know, does it worth it? Do you know how to code in NodeJs or Java and wish to see this script takes another turn and provide another languages support? I'll be thrilled if that happened.

Stay Safe.