Serverless Docker Patterns

ℹ️ Please read our new post New Amazon Linux Dev Container Features for our latest patterns.

This is the first part in a series on how we use Docker for Rails on AWS Lambda at Custom Ink. After reading, drop a comment and share how you or your teams leverage Docker & Serverless together. Did you find any of these tips useful? Perhaps you use the serverless framework instead of the AWS SAM CLI tooling? If so, how does that work for you?

This post will focus on three abstract patterns derived from our usage of the docker-lambda images with AWS SAM for Rails projects. My hope is that you can apply these to your own specific Docker work.

1. Normalized Script Conventions
2. Cross-Platform SSH Agent for CI/CD
3. Mac OS Filesystem Performance

Normalized Script Conventions

Long before our adoption of Docker, every Custom Ink project required the Scripts to Rule Them All pattern which allows any engineer to get "up and running" with any project quickly. Our more modern Docker usage compliments this pattern with a few small changes that make sense for us:

• We use the bin directory vs the script directory.
• Because we use Docker/Compose, there is no need for ci specific scripts.
• Since we are primarily a Rails shop, we have a bin/console for easy REPL access.
• We extended this pattern to include bin/deploy for cloud native projects like Lambda.
• A bin/update script that will tear down any previous resources then bootstrap & setup.

Because we make heavy use of docker-compose which helps us abstract out common environment variables, shared volumes, and services, all of these scripts will use docker-compose run vs docker run. This avoids duplicating CLI args across every bin script, but it also has the added benefit of making your usage of Docker mirror real local development. A few details and outcomes include:

1. Every bin/${name} script is a simple docker-compose run bin/_${name} wrapper. Avoids tons of compose args and escaping issues. Helps longer scripts like setup to be more maintainable.
2. We have essentially commoditized CI/CD. Long gone are the special "how do I get this project running" on TravisCI, CircleCI, or GitHub Actions. Any system with Docker & Git will "just work", promise!
3. No ENTRYPOINT or CMD means bin/server takes on the responsibility of passing ports and delegating to your projects native local server. This allows other bin script like running one off tasks or console access all avoid starting an superfluous server process.

Curious what this looks like? Our Lamby Quick Start leverages an AWS Lambda cookiecutter project that adopts both these script conventions and docker-compose patterns. Give it a go and deploy a starter Rails application to Lambda or have a look at the code if you want.

Cross-Platform SSH Agent for CI/CD

Remember that promise I made on the commoditization of CI/CD because all local development & testing happens with Docker? There is one gotcha. Access to private Ruby gems or Node packages hosted on GitHub. A very common use case for us at Custom Ink and I suspect many large companies.

For a long time we used GitHub access tokens. But this required special platform-specific tooling for both local development and our CI/CD pipelines. For Node this often required crazy hacks to the package.json file too. The internet is full of posts on how to solve this. All of them however speak to a specific platform or language's package manager. Could there be a unified SSH pattern? Yes! Here is what I found works.

In your docker-compose.yml file under your application or service, add or merge these environment/volume settings below.

environment:
  - SSH_AUTH_SOCK=${SSH_AUTH_SOCK}
volumes:
  - ${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}
  - ~/.ssh/known_hosts:/root/.ssh/known_hosts

In order for it to work on both Linux & Mac, you must setup the SSH_AUTH_SOCK environment variable to the specific value required by OS X. Since your bin/_setup script will most likely be calling either bundle or yarn install, we add this bash condition right before our docker compose run command.

if [[ "$OSTYPE" == *"darwin"* ]]; then
  export SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock
fi

docker-compose run myapp ./bin/_setup

We love GitHub Actions at Custom Ink and leverage it for all of our AWS Lambda CI/CD pipelines. Thanks to organization secrets and great marketplace additions like webfactory/ssh-agent, we found the above patterns worked with no fuss by adding this to our workflow before the setup step.

- name: SSH Agent
  uses: webfactory/ssh-agent@v0.4.0
  with:
    ssh-private-key: ${{ secrets.MYORG_GITHUB_SSH_KEY }}

Remember, that this pattern should work for any language's package manager that can leverage SSH access to private GitHub packages. If you are using Ruby's bundler, make sure your use this github source format in your Gemfile.

git_source(:github) { |repo| "git@github.com:#{repo}.git" }

For Node's yarn (maybe npm too), your dependencies would look like this in package.json.

"image_changer": "github:myorg/myproject.git"

Mac OS Filesystem Performance

I never like to speak poorly of anyone's or company's work. But I think we can all admit that Docker Desktop for Mac with large projects has downright horrible filesystem performance. It has been this way for years and as such the internet is littered with solutions and hacks.

Part of me would like to believe that a Twitter ❤️ by the VP of Product for Docker might mean they can be trusted to reverse course on trying to, yet again, write their own file system. We can all hope. But in the meantime we need Docker unlocked for development for our predominantly MacBook driven engineering teams. But how can we do this with the following value questions:

1. Embrace hope and plan for the spacklepunch?
2. Be minimally invasive and easy to delete?
3. Work cross-platform from local development to CI/CD pipelines?

There are tons of solutions out there from Docker Machine to dinghy. After some careful research I found docker-sync was the easiest to adopt while addressing the concerns above. Here is how we added docker-sync to both our Lambda & Container Rails projects.

First, we created or added this line to our projects .env file which leverages compose's overlay capability. By doing this we ensure every bin script that uses docker-compose run remains untouched.

COMPOSE_FILE=docker-compose.yml:docker-compose-dev.yml

Assuming your docker-compose.yml has a service named myapp, your newly created docker-compose-dev.yml overlay file will look like this. Remember, our bin/server takes responsibility for our entry point. Since technically compose up needs a service, we have added a non operation tail command here. If your service already has an entry point, you can omit this line.

version: '3.7'
services:
  myapp:
    command: tail -f /dev/null
    volumes:
      - myapp-sync:/var/task:nocopy
volumes:
  myapp-sync:
    external: true

Finally we add our simple docker-sync config file that connects up the myapp sync along with setting our Docker scope to the present working directory. Here is our docker-sync.yml file:

version: '2'
syncs:
  myapp-sync:
    src: '.'

Finally we need every other bin script like console, server, and friends to just work. To do this we need to start our services so each docker-compose run myapp command works as it did before without docker-sync. We also need to do this in a platform agnostic way. Thankfully, our bin/setup bash check for OSTYPE is the perfect place to add our new docker-sync logic.

if [[ "$OSTYPE" == *"darwin"* ]]; then
  export SSH_AUTH_SOCK=/run/host-services/ssh-auth.sock
  gem install docker-sync
  docker-sync start
  docker-compose up --detach
fi

docker-compose run myapp ./bin/_setup

That's all 🎉🎉🎉. With only a few additions everything else in our Docker project remains untouched. If you are on a Mac, you will get close to native speed after the initial file sync setup has run. If the Docker team (bless their hearts) ever ships a usable and performant filesystem, we can delete a few files and call it a day. Here are some additional things to consider when adopting this pattern:

1. Make sure different projects use a unique myapp naming convention. Failing to do so could mean two projects share the same docker-sync file system. Don't cross the streams!
2. For your team members that use Linux or for your CI/CD system which presumable also uses Linux, remove the .env file or override the COMPOSE_FILE environment variable to only have the single docker-compose.yml file. Hence, avoid the overlay.
3. Remember to add .docker-sync* to your .gitignore file.
4. Consider adding a few helpers to your bin/update script to cleanup docker-sync. Example:

docker-compose down
docker-sync stop
docker-sync clean

./bin/bootstrap
./bin/setup

Thanks for Reading!

As they become available please check out my other posts on this series. And remember, I would love to hear how you are using Docker for Serverless or Containers on your projects. Thanks!!!