Building Static Ruby Gems for Lambda Zip or Containers using Docker & Amazon Linux

Welcome to my 3rd post in a long running series on Lambda & Containers. If you are a Rubyist new to Lambda and the Serverless Application Model (SAM), consider looking over this Ruby Microservice Workshop.

However, if you have been using Ruby and/or Rails with AWS Lambda, today I would like to talk about a few patterns on how to use Ruby gems with non-standard system dependencies with AWS Lambda. Typically, this problem had two solutions.

1️⃣ Lambda Layers

Prior to Lambda Containers we had to build a Lambda Layer using a myriad of techniques. The result was always something in the /opt directory, yet finding the right balance of what to put in that directory has always been a pain. For example, if your gem uses FFI (like ruby-vips) how do I build a layer, test it, and make sure that same layer is available for local development? Often times this required engineering some Docker & AWS SDK coordination. Not a simple process when means it could easily break or slow down testing upgrades to either the gem or supporting layer.

2️⃣ Lambda Containers

Shortly after its release in December, we upgraded Lamby (Simple Rails & AWS Lambda Integration) to use containers as our default package method. Our project starter uses docker-compose for local development. This allows us to separate our "build" image for development & testing from our production "runtime" image by using two distinct Dockerfiles. If your project needed a new dependency, the solution was simple. Add a few yum installs in both files and ship the resulting image. What could be easier?

RUN yum install -y \
    pkgconfig \
    libpng \
    librsvg2

If a package was not available to install via yum, you could just as easily compile or install it form source another source.

Potential Issues?

I think most would agree that when it comes to shipping code fast, the container deployment package for Lambda is far easier to reason with. Depending on your needs, this solution is likely just fine. But what might be some drawbacks?

First could be the image size. Lambda does allow up to 10GB for a container deployment package, but you are going to want to keep that image as small as possible. Smaller images means faster deploys and quicker start up times. Often Ruby gems will only need a small subset of headers or shared objects to link against. Meaning image bloat is easy when reaching for yum or manual installs.

Related to the above, another issue could be image security. Thanks to ECR's Image Scanning, we can see warnings for our pushed images. Often times when installing non-packaged system dependencies we end up with artifacts that are considered insecure. Like an older gcc or cmake needed to build a dependency. Patterns like Docker's multi-stage builds are one way of dealing with these issues. But is there a 3rd option for some Ruby gems?

3️⃣ Precompiled Gems

I am a huge fan of Node and we use it a lot at Custom Ink. Popular npm libraries, like Sharp image processing, come with prebuilt binaries which avoids the need for developers to install system dependencies ahead of time.

The Ruby community is no stranger to this practice. Thanks to Ruby Heros like Luis Lavena, many of our popular gems have precompile Windows binaries. Today other projects like Nokogiri are now supporting precompiled gem packages for several platforms. This makes gem installs easier for both Lambda zip and containers by avoiding layers or container deployed system dependencies. It also makes our bundle installs much much faster. This time adds up for both development and our build pipelines.

Short of asking our beloved gem maintainers to do the hard work, how can we adopt these patterns for gems we use in our own Lambda projects? This is what I wanted to explore.

Precompiled MySQL2 Gem

We use MySQL via AWS Aurora for both small and large applications at Custom Ink. Prior to Lambda Containers, I made this project to build a precompiled mysql2 gem with statically linked libmysqlclient to avoid Lambda Layer development pains. But we have found it really helps with Lambda Containers too.

https://github.com/customink/mysql2-lambda

gem 'mysql2-lambda'

The gem might as well be named mysql2-amazon-linux or as the mysql2 with x86_64-linux platform support since it should work for any matching one.

For this post I'd like to share how I built this project. If you like it, maybe you can apply some or all the techniques to your own Ruby database gems like pg, tiny_tds, or ruby-oci8. Who have yet to offer these via their official gem sources.

DIY Precompiled Gems

Here are some of the highlights from the mysql2-lambda project on how I built this gem for Lambda.

Docker

You will need to know Docker well. From passing arguments, mounting volumes to share files, and running commands.

Pick a consistent image name to build. Doing so means you can "shell in" to the container to debug if needed. I use this function dk-lbash all the time.

dk-lbash () {
    image=$1
    docker run \
    --interactive \
    --tty \
    --rm \
    --volume "${PWD}:/var/task" \
    --entrypoint "/bin/bash" 
    $image
}

Base Image

Since this project is for Lambda, we used the base image provided by AWS SAM CLI.

FROM amazon/aws-sam-cli-build-image-ruby2.5

This image closely resembles the Lambda runtime but it also has additional build tools useful to get the gem built. The final gem package works for both Ruby 2.5 and 2.7 and likely any Amazon Linux.

Upstream Gem

Each gem will have different ways to install it. Remember to read their documentation. For example, the mysql2 gem had built in support to statically link the system dependencies using the --with-mysql-dir option. Look for ways you can statically link dependencies.

Linking Dependencies

Our goal is to have all the dependencies linked in the gem's shared object .so file(s). For example, this is the resulting mysql2 installed we needed to check and repackage.

/var/lang/lib/ruby/gems/2.5.0/gems/mysql2-0.5.3/lib/mysql2/mysql2.so

There is a great tool called ldd which lists dynamic dependencies. In my case the mysql2 gem did not statically link all dependencies. This is why I had to use patchelf to fix a few.

Testing

After we install, copy out our new gem files, and use gem build, we should at least test to make sure it works before pushing to RubyGems. For this task I used docker-compose. The full details are in the project, but this example below shows a compose file that mixes the official Lambda runtime image with a MySQL service to run a test file.

version: "3.8"
services:
  test2.5:
    image: public.ecr.aws/lambda/ruby:2.5
    entrypoint: ./bin/_test
    volumes:
      - $PWD:/var/task
    depends_on:
      - mysql
    links:
      - mysql
  mysql:
    image: mysql:5.7
    environment:
      - MYSQL_ROOT_PASSWORD=root

Thanks

Hope you found this useful. I'd love to hear your thoughts on this post and if anyone is working on precompiled gems for other popular database gems?