Add native binaries to your Nodejs Lambda functions

Written by Martin Wentzel

You sometimes want to call a native binary from your lambda code. This can be something as simple as ls or more complex like git or imagemagick. Those binaries are not available in Lambdas by default and I looked for a way to add them.

I am currently working on a project where I need to issue git commands to clone a repo from Github; this is not achievable with the Github API.

It is possible to build and deploy a lambda with docker, but building lambdas in this way does not give us the possibility to install binaries. The base images amazon/aws-lambda-nodejs do not even have a proper bash installed, let alone any kind of package manager.

Solution

In order for docker image to be compatible and work on Lambda, it needs to have an Runtime Interface Client(RIC), which receives requests from and sends requests to the Lambda service. You can simply use such an RIC as the entrypoint of your docker image and you should be good to go.

You can find more technical details about RICs and the Lambda Runtime API under https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html

There is already an RIC for Nodejs. The README also contains an example for a multi-stage build that gives you a docker image, which is ready to be executed as a lambda function.

In the first build step they have a build-image stage to install all dependencies and install the aws-lambda-ric package. It is worth noting that installing the aws-lambda-ric package still comes with a compile step, as the package has some native dependencies which need to be compiled. For the second build stage of the Dockerfile, all needed files are copied over to a new image. This gives us a smaller final image, which does not need to contain build dependencies (this is also one of the best practises of writing Dockerfiles)

We can simply utilize the second build block to add our own binary. In our example we are going for git:

FROM node:12-buster-slim

### Block below is new
################
RUN apt-get update && \
    apt-get install -y \
    git && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

### new block finished
################

# Include global arg in this stage of the build
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}

# Copy in the built dependencies
COPY --from=build-image ${FUNCTION_DIR} ${FUNCTION_DIR}
ENTRYPOINT ["/usr/local/bin/npx", "aws-lambda-ric"]
CMD ["app.handler"]

That way we can finally call git clone from our handler function inside app.js

exports.handler = async (event, context) => {
    execSync(`git clone --depth 1 https://GithubPatToken@github.com/docker/docker`)
    return 'Successfully cloned github.com/docker/docker!';
}

and succesfully clone a repository.

Optimizations

I have found that the aws-lambda-ric package and the example Dockerfile have a few shortcomings and can be optimized even further:

The resulting size of the docker image is quite large and with a few tricks we can shave off a few hundred MBs
The aws-lambda-ric package has some compile steps upon running npm install, which takes almost 2 minutes to complete. Additionally, it only supports Nodejs up until v14 which, according to the release schedule, is almost at the end of maintenance and 4 major versions old.

Size Problem

The whole src folder is copied and npm install installs all dependencies of the package.json of your project.

Solution

You should bundle your lambda source code with esbuild. That way you have one standalone file that already contains all dependencies. And it also works with Typescript. An example command would be:

npx esbuild  ./src/your_entrypoint_file.ts \
    --bundle --target=node14 \
    --outfile=index.js  --platform=node

This creates one single index.js file containing all of your application code.

Than you can optimize the costly parts of the first build-image stage to

# Define custom function directory
ARG FUNCTION_DIR="/function"

FROM node:12-buster as build-image

# Include global arg in this stage of the build
ARG FUNCTION_DIR

# Install aws-lambda-cpp build dependencies
RUN apt-get update && \
apt-get install -y \
g++ \
make \
cmake \
unzip \
libcurl4-openssl-dev

# Copy function code
RUN mkdir -p ${FUNCTION_DIR}
COPY index.js ${FUNCTION_DIR} # this is changed

WORKDIR ${FUNCTION_DIR}

RUN npm install aws-lambda-ric # and this changed as well

For my project this shaved more than 200MB off the final image:

With a full npm install and copying all dependencies the final size was 529 MB
Using esbuild and only installing aws-lambda-ric, the final size was 264 MB, a reduction of 50%!
Problem with compiling and only supporting up until Node v14

This is something that I will optimize and tackle in a future blog post.

Conclusion and Outlook
We have learned how to add and pack native binaries inside our lambda function. There still is a lot of potential for optimizations as the resulting docker image is still 264MB large. In the next blogpost we will take a look at how to optimize the bundling and installing of aws-lambda-ric, and maybe even get it to compile on an alpine based image so that we can use node-alpine images.