Run local Graviton2 builds with AWS CodeBuild agent

Previously I described how to use AWS CodeBuild on Graviton2. The example I used was very simple. It required only Docker to execute the build and did not run for a long time. For more complex projects it is helpful to have an AWS CodeBuild environment on your local machine. This allows the Docker images to be inspected to make sure they contain the needed tools. It also provides an easier way to work through the build process. New users will likely need some practice to debug the contents of the buildspec.yml file. This article explains how to run builds locally with the CodeBuild agent and create CodeBuild projects for Graviton2.

Setup and prerequisites

AWS EC2 instances powered by Graviton2 processors use the Arm architecture. A local machine which also uses the Arm architecture provides a seamless developer experience. Today, I’m going to do it on a MacBook Air with Apple Silicon. Windows 10 on Arm is another option.

Docker and git are required to run this tutorial. Git should be available in /usr/bin already. Install Docker for Apple Silicon using the Download link.

With git and Docker available, let’s setup the required Docker images.

Docker images

There are two Docker images required to run CodeBuild. The build image and the local agent. The build image is the one where the build process will be executed. It needs to have all of the required tools to run the commands in the buildspec.yml file for the project.

The Dockerfiles for CodeBuild images are available on GitHub. For the previous article I selected the Amazon Linux 2 standard image, version 2.

To build and use any of the CodeBuild images clone the GitHub repository, change directory to the desired image, and build.

I have posted the Amazon Linux 2 image for Arm on Amazon ECR Public so there is no need to build it.

To get the build image use docker pull.

$ docker pull public.ecr.aws/z9p7l6s8/codebuild/amazonlinux2-aarch64-standard:2.0
$ docker tag public.ecr.aws/z9p7l6s8/codebuild/amazonlinux2-aarch64-standard:2.0 codebuild/amazonlinux2-aarch64-standard:2.0

To get and build yourself (or any other CodeBuild image) use docker build.

$ git clone https://github.com/aws/aws-codebuild-docker-images.git
$ cd aws-codebuild-docker-images/al2/aarch64/standard/2.0
$ docker build -t  codebuild/amazonlinux2-aarch64-standard:2.0  .

The second Docker image needed is the local agent. This is now available from Docker Hub for Graviton2. This is good news as it wasn’t available when I wrote the previous article.

Use docker pull to get the local agent image.

$ docker pull amazon/aws-codebuild-local:aarch64 --disable-content-trust=false

CodeBuild local build script

The last thing required is the script which runs the build. The script is available in the GitHub project above which contains the Dockerfiles, but it can also be downloaded using wget.

$ wget https://raw.githubusercontent.com/aws/aws-codebuild-docker-images/master/local_builds/codebuild_build.sh
$ chmod +x codebuild_build.sh

Running a build

A test build can be started using the buildspec.yml. In my previous article I put the buildspec.yml into the GitHub project. Sometimes users may also enter the buildspec.yml directly into CodeBuild. Either way the build script defaults to a buildspec.yml file in the directory where the script is run. It also offers a switch to specify the buildspec.yml file.

Let’s run a local build using the same “hello world” project.

Clone the project from GitHub to your local machine.

$ git clone https://github.com/jasonrandrews/hello-arm.git
$ cd hello-arm

The buildspec.yml is already in the top level directory.

One other thing to note is the use of the secrets-manager section in the buildspec.yml. AWS services such as Secrets Manager can be used on the local machine. Setup the AWS cli on the local machine to create a profile which has access to your AWS account and use the -c option to codebuild_build.sh.

Here is the command to run the build:

$ ./codebuild_build.sh -c -i codebuild/amazonlinux2-aarch64-standard:2.0 -a /tmp  -l amazon/aws-codebuild-local:aarch64

Checking the CodeBuild images

Another common reason for running CodeBuild locally is to make sure everything needed by a build is contained in the Docker image.

To start the CodeBuild image and poke around to see what is inside of it override the ENTRYPOINT from the Dockerfile and start bash instead.

$ docker run --rm -it --entrypoint /bin/bash codebuild/amazonlinux2-aarch64-standard:2.0

This allows you to look and see what is inside the image, try out commands interactively, and gain confidence the image will be able to run the buildspec.yml.

Use the -h on codebuild_build.sh to get a list of more options for the build script.

 $ ./codebuild_build.sh -h
usage: codebuild_build.sh [-i image_name] [-a artifact_output_directory] [options]
Required:
  -i        Used to specify the customer build container image.
  -a        Used to specify an artifact output directory.
Options:
  -l IMAGE  Used to override the default local agent image.
  -r        Used to specify a report output directory.
  -s        Used to specify source information. Defaults to the current working directory for primary source.
               * First (-s) is for primary source
               * Use additional (-s) in <sourceIdentifier>:<sourceLocation> format for secondary source
               * For sourceIdentifier, use a value that is fewer than 128 characters and contains only alphanumeric characters and underscores
  -c        Use the AWS configuration and credentials from your local host. This includes ~/.aws and any AWS_* environment variables.
  -p        Used to specify the AWS CLI Profile.
  -b FILE   Used to specify a buildspec override file. Defaults to buildspec.yml in the source directory.
  -m        Used to mount the source directory to the customer build container directly.
  -d        Used to run the build container in docker privileged mode.
  -e FILE   Used to specify a file containing environment variables.
            (-e) File format expectations:
               * Each line is in VAR=VAL format
               * Lines beginning with # are processed as comments and ignored
               * Blank lines are ignored
               * File can be of type .env or .txt
               * There is no special handling of quotation marks, meaning they will be part of the VAL

Summary

AWS CodeBuild is a great service to automate most anything and works well on Graviton2. With the newly available CodeBuild local agent Docker image for AArch64 builds can be developed and tested locally to quickly find any issues. Local testing can be done on a MacBook with Apple Silicon or a Windows 10 on Arm machine such as the Samsung Galaxy Book S or the Microsoft Surface Pro X.