Using Docker manifest to create multi-arch images on AWS Graviton processors

Docker manifest is an experimental command to build docker images which support multiple architectures. Let's see how it works and compare it to docker buildx.

In 2019, docker buildx was created as a way to build multi-architecture images. At the time, developers didn't have Arm in the cloud or on their desk. Projects were interested in the Arm architecture, but as a secondary option for AWS Graviton or even Raspberry Pi. With buildx, a single build command creates a multi-architecture image. In some cases, an Arm machine is not even needed for the build.

Today, developers are using the Arm architecture on their desk and in the cloud. The trailblazing success of AWS Graviton processors has motivated developers to try out Graviton for themselves. The popularity of Mac computers with Apple silicon means many developers have the same architecture on their desk. Today, I will explain how to build multi-arch images using the docker manifest command. I have found this to be the easiest way to create multi-arch images for many projects.

Background

Docker buildx provides a build command to create images for multiple architectures. I’m primarily using arm64 and amd64. If you use other architectures, the concept can easily be extended.

The challenge with buildx is that it uses emulation, cross-compilation, or a remote builder for the target architectures which do not match the build machine. Emulation works fine for building images which primarily copy and install files, but it can suffer from functional issues or performance problems for more complex image building. Sometimes cross-compilation is easy, but sometimes it is difficult. Using a remote builder is not convenient for testing changes. Changing the remote builder setup while testing can be confusing.

Docker buildx also confuses developers who are expecting a local image to immediately run after a successful build. Because buildx targets multiple architectures, it must be saved to a repository, not local storage. This requires a push to a repository and then a pull to run. There are some other workarounds, but the use model is different from what many developers expect. Searching Stack Overflow and GitHub confirms that a missing image after successfully running buildx is a common question.

I spend the majority of my time on the Arm architecture. A shrinking number of projects require amd64 support, but when they do, I find myself shying away from buildx. Docker buildx is one way to approach the challenges of multiple architectures, but docker manifest feels easier for me.

Let's see how it works.

Docker manifest

Docker includes an experimental feature called docker manifest. Please be aware the feature is experimental and not recommended for production use.

Docker manifest provides a different way of working. It allows separate images to be built for each architecture, which can be joined into a multi-arch image when it's time to share. This enables me to build and test on any architecture and postpone using multi-arch until later. When it's time to share a multi-arch image, it can be created in seconds using docker manifest. Docker manifest also makes it easy to update one of the architectures without emulation or remote builders.

Let’s do a quick comparison between buildx and docker manifest.

Here is a small example Dockerfile.

FROM alpine AS builder
RUN apk add build-base
WORKDIR /home
COPY hello.c .
RUN gcc "-DARCH=\"`uname -a`\"" hello.c -o hello

FROM alpine
WORKDIR /home
COPY --from=builder /home/hello .
CMD ["./hello"]

The Dockerfile compiles and runs a hello world C program.

#include <stdio.h>
#include <stdlib.h>

#ifndef ARCH
#define ARCH "Undefined"
#endif

int main()
{
    printf("Hello Arm Developers, architecture from uname is %s\n", ARCH);

    switch (sizeof(void *))
    {
        case 4:
            printf("32-bit userspace\n");
            break;
        case 8:
            printf("64-bit userspace\n");
            break;
        default:
            printf("unknown userspace\n");
    }
    exit(0);
}

Copy the Dockerfile and hello.c to your computer for the steps below.

Let’s see how to use it with buildx to build a multi-architecture image.

Docker buildx

Using buildx involves setting up a builder and running the buildx command.

Below is a script showing how to use buildx. Copy the file, change the HUBU variable to your Docker Hub username, and run the script to try it.

#!/bin/bash

HUBU=hubuser
IMG=hello-world

docker buildx create --name mybuilder
docker buildx use mybuilder
docker buildx build --platform linux/amd64,linux/arm64 -t $HUBU/$IMG --push .

The script automatically pushes the multi-arch image to Docker Hub. Without the push argument, the results will not be available. It’s easy enough to pull the image and run it, but with a larger image it does take time to push to the registry and pull it back.

There are numerous tutorials about using buildx if you want to learn more.

Docker manifest

The docker manifest command is useful to combine multiple existing images into a single multi-architecture image.

To keep track of the images, I use the image tag to indicate the architecture.

To build the same Dockerfile without buildx use this command. This produces a single image, just for the architecture of the current machine.

docker build -t hello-world:$(arch) .

The image is available locally to run. On AWS Graviton, the tag will be aarch64 to differentiate it from the same command run on an x86_64 machine.

After building an image on each machine you want to support, tag the images for a Docker Hub account and push. Again, change to your Docker Hub username to try it.

docker tag hello-world:$(arch) jasonrandrews/hello-world:$(arch)
docker push jasonrandrews/hello-world:$(arch)

The image below shows both tags in Docker Hub.

The last step is to join the two tags into a single multi-architecture image.

docker manifest create jasonrandrews/hello-world:latest \
--amend jasonrandrews/hello-world:aarch64 \
--amend jasonrandrews/hello-world:x86_64

docker manifest push --purge jasonrandrews/hello-world:latest

The purge option is not needed the first time, but I found that to update one of the images and update the multi-arch image it was needed.

After the manifest push, a new multi-arch image with the latest tag is available. Pulling hello-world:latest from either architecture now works, and a user doesn't need to pay attention to the computer they are using.

Summary

The experimental docker manifest command offers a way to create multi-architecture images by joining multiple images using a manifest. Docker manifest provides additional features not covered here, and is useful to create, inspect, and modify manifest lists. As the docker manifest command matures and developers increase AWS Graviton usage, it provides another way to build docker images.

Give it a try and see how it works.