Building Secure Docker Images for Production - Best Practices

With modern applications, Docker is a common tool on the tool belt.
We use Docker to develop and test our application and to run it in production.

Have you ever thought about the security of Docker containers in production?

While Container security is a wide field, let's focus on building more secure images for production deployment without spending hours online researching best practices.
This blog post introduces a low-friction approach to finding potential vulnerabilities in your images.

A basic development image

Let's say we're working on an API service written in Rust. The repository contains a Dockerfile with just a handful of instructions. It might look like this:

FROM rust:1.70
EXPOSE 8000
COPY ./ ./
RUN cargo build --release
CMD ["./target/release/app"]

At first glance, the file looks straightforward. Docker Hub offers an official image for almost every programming language. The official Rust image has a full toolchain preinstalled. We can use this image right away to compile and run the project—a perfect fit for a development environment.
As soon as we go into production, however, things look different.

Production requirements

We don't need the compiler toolchain when working with a compiled language. The Rust binary is self-contained, only requiring a libc runtime (which often is already present).
We might consider only shipping a minimized bundle to reduce the image size for a language like JavaScript.

No matter which language we work with, a production image must satisfy different requirements than a development image, such as installing specific package versions, strict permissions, etc.

Another aspect is image size. A development image can easily take up lots of disk space. Consider this example where an image based on rust:1.70 takes up 3GB:

$ docker images
REPOSITORY                                                                  TAG            IMAGE ID       CREATED          SIZE
rust-dev                                                                latest         3ad31037e881   26 seconds ago   3.07GB

The compiler toolchain eats up some of the space, but the build artifact folder is most likely responsible for most of it. We need neither to run the application in production.

Building a production-ready Docker image

Let's optimize our Dockerfile. To reduce image size, we implement a multi-stage build. With this approach, we still use the official Rust image to build the application. Once compiled, we copy the binary out of it into a small production image that only has a few things installed and configured. These changes lead to a much smaller disk space footprint.

FROM rust:1.70.0 as builder

WORKDIR /usr/app
RUN USER=root cargo new --bin pokemon_api
WORKDIR /usr/app/pokemon_api

COPY ./Cargo.toml  ./Cargo.toml

RUN cargo build --release
RUN rm src/*.rs
COPY ./src ./src

# 5. Build for release.
RUN rm ./target/release/deps/pokemon_api*
RUN cargo build --release
#--------
FROM debian:bookworm-slim
EXPOSE 8000
RUN apt-get update
RUN apt-get install -y curl && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/app
COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api
ADD ./rocket_config.toml Rocket.toml
CMD ["/usr/app/pokemon_api"]

With our new production image in place, let's find out if there's more we can optimize.

Prerequisities

In the following steps, we use a local Kubernetes cluster (such as kind) to test the image.
With the cluster up and running, let's install some tooling to help us with image scanning.
In this case, we're using KubeClarity. Follow the installation instructions in the README to install it into your development cluster.

Deploy Docker image to a local cluster

With KubeClarity installed, let's deploy our service.
(Find the example source code here.)

kubectl create ns pokemon
kubectl apply -n pokemon -f k8s/deployment.yaml

As mentioned, this service is written in Rust, performing a simple HTTP Call to fetch a Pokemon CSV, which returns JSON data.
Upon further inspection of the source code, you'll notice it uses std::process::Command and curl to perform the HTTP request.
With this, we're simulating the code's dependency on specific system packages (curl).

Before we deploy this service in other environments, such as staging or production, we want to find out if there are any potential security problems with this particular image.

Scan with KubeClarity

If you haven't already, in a separate terminal window, run the following command:

kubectl port-forward -n kubeclarity svc/kubeclarity-kubeclarity 9999:8080

Then, head over to http://localhost:9999/runtimeScan.

On the top, select pokemon as namespace and click on Options to open the settings dialogue.

In the settings dialogue, select CIS Docker Benchmark.

Save your changes and click Start Scan.

After a few seconds, the scan will conclude and show us a summary of the findings.

Head to http://localhost:9999/applicationResources, click on ghcr.io/schultyy/rust-pokemon-api:0.0.1 in the list.
On the detail page, select CIS Docker Benchmark.

The list shows seven findings with varying severities.

Explore findings

Clear apt-get caches

This step reduces the image size and cleans up superfluous and unneeded caches.
Check out this list for more details.

Use COPY instead of ADD in Dockerfile

COPY and ADD both have an overlap in features. Prefer to use COPY over ADD as COPY only supports basic file-copying mechanisms.
ADD, however, has additional features that are not immediately obvious, such as tar extraction or remote URL support.

See docs.docker.com for more details.

Do not use update instructions alone in Dockerfile

By combining apt-get update with apt-get install, Docker will cache the update layer and reduce the total number of layers.

Consider this example:

FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install package-a

In this case, the first time you run docker build, it executes every command and caches every line. The next time you run docker build, Docker determines nothing has changed and will finish much faster.

You return to this Dockerfile a few days later, realizing you need to install an additional package.

FROM ubuntu:22.04
RUN apt-get update
RUN apt-get install package-a package-b

Upon build, Docker realizes the first line hasn't changed and will use a cached layer. Distributions like Debian and Ubuntu update repositories frequently (i.e. deleting old package versions).
Package sources updated by apt-get update from two days ago might point to a package or version that no longer exists, causing apt-get install to fail.

If you combine both lines, however, you will get updated package sources every time the list of packages to install changes.

More details

Create a user for the container

Running your application as root comes with several security implications. Therefore, dropping privileges as soon as possible is a good practice.

RUN groupadd -g 10001 appuser && \
   useradd -u 10000 -g appuser appuser \
   && chown -R appuser:appuser /app

USER appuser:appuser

All instructions after USER will run as appuser, without privileges.

More details:

• CIS-DI-0001
• docs.docker.com

HealthCheck

It's a good practice to include a Health Check within the Dockerfile to allow Docker to determine if the container is healthy.

This addition is helpful when the container is running but the application within the container has crashed.

HEALTHCHECK CMD curl --fail http://localhost:3000 || exit 1

See docs.docker.com for more information.

Implement fixes

With all discussed findings implemented, our Dockerfile now looks like this:

FROM rust:1.70.0 as builder

WORKDIR /usr/app
RUN USER=root cargo new --bin pokemon_api
WORKDIR /usr/app/pokemon_api

COPY ./Cargo.toml  ./Cargo.toml

RUN cargo build --release
RUN rm src/*.rs
COPY ./src ./src

# 5. Build for release.
RUN rm ./target/release/deps/pokemon_api*
RUN cargo build --release
#--------
FROM debian:bookworm-slim
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 CMD [ "curl --fail http://localhost:8000/health" ]

RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*
WORKDIR /usr/app
COPY --from=builder /usr/app/pokemon_api/target/release/pokemon_api /usr/app/pokemon_api
COPY ./rocket_config.toml Rocket.toml

RUN groupadd -g 10001 appuser && \
   useradd -u 10000 -g appuser appuser \
   && chown -R appuser:appuser /usr/app

USER appuser:appuser

CMD ["/usr/app/pokemon_api"]

Next, we build a new image version:

docker build -t ghcr.io/schultyy/rust-pokemon-api:0.0.2 .

Once built, let's push it to the registry:

docker push ghcr.io/schultyy/rust-pokemon-api:0.0.2

With the new image version published to the container registry, let's deploy it into the test cluster:

kubectl apply -f k8s/deployment.yaml

As a last step, we re-run the scan (with the steps above) to verify we have no more fatal findings:

Next Steps

Performing these kinds of checks is only of KubeClarity's features. Whenever you run a scan, it also scans for vulnerable packages. If you want to learn more about package scanning, check out this blog post.

Also, make sure to check out the KubeClarity GitHub project!