At Wehkamp we've been using ASP.NET for a long time. Containers brought .NET with us to our microservices architecture. We used a shell script in a special build container to build our service. Today we'll discuss how we turned that shell script into a multi stage build Dockerfile.
- Intro
- Goals
- Project structure
- Step 1: Configuration with global arguments
- Step 2: Build stage
- Step 3: runtime stage
- All together
- Reading list
- Changelog
- Comments
Goals
With the new setup we have some goals in mind:
-
Reproducibility — it should be on my machine as it is in the pipeline. It would be great if I can just do a
docker build .
and everything should start building as it does in Jenkins. - Cacheability — Docker uses layers, which greatly speeds up the system. When we do a restore as a set of layers, we don't need to generate network traffic repeatedly.
- Upgradability — the current solution will focus on .NET 7.0, but we need things to be easily upgradable to a newer (or older) version of .NET.
- Readability — the Dockerfile should be treaded as a DevOps manifest, showing how to build and run our application. Let's make sure people understand what's going on. But also the output that Docker shows in the terminal should be easy to read and understand.
If at some point we can add security to the Dockerfile, that would also be great.
Project structure
Most of what we do is based on convention. Our project structure reflects those conventions:
.
├── src/
│ └── Blaze.Prime.Service.Api/
│ ├── Blaze.Prime.Service.Api.csproj
│ └── ...
├── test/
│ └── Blaze.Prime.Service.Api.UnitTests/
│ ├── Blaze.Prime.Service.Api.UnitTests.csproj
│ └── ...
├── .dockerignore
├── .gitignore
├── Blaze.Prime.sln
├── Dockerfile
└── nuget.config
It is important that the name of the C# project directory matches the name of the C# project file.
Step 1: Configuration with global arguments
Our Dockerfile may grow. Let's make it easy to upgrade and changing global settings by using ARG instructions. We will use these arguments in the build and runtime images.
########################
# Configuration settings
########################
ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
API_DLL="$API_NAME.dll" \
PORT=5000 \
ASPNET_VERSION="7.0" \
# options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
# minimal keeps your pipeline readable while inforing you what's going on
VERBOSITY="minimal" \
APP_DIR="/app" \
TAG="" \
EXECUTE_TESTS="true"
Let's construct the project and DLL names by convention. To make debugging easier, we set the VERBOSITY
argument centrally. We find that the minimal
settings is a great default, as it just gives enough output to understand what's going on.
Our CI fills the TAG with the tag of the Docker image that is being built.
Step 2: Build stage
Now that we have the arguments in place, let's start with the beginning of our build image. We use the standard SDK provided by Microsoft. We don't feel the need to share telemetry and we would like to have our times in local Amsterdam time.
################################################
# Build image, uses ASPNET_VERSION, API_PROJECT,
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################
FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build
ARG API_PROJECT VERBOSITY APP_DIR
ENV \
TZ=Europe/Amsterdam \
DOTNET_CLI_TELEMETRY_OPTOUT=1 \
DOTNET_NOLOGO=0
WORKDIR /build
Notice how we're not defining TAG
and EXECUTE_TESTS
here. If we would, we would lose our caching ability when a new tag is used or when testing is switched off.
2.1 Restore in cacheable layers
In our old script we just executed a dotnet restore
as part of the build script, which is fine, but nothing is cached by Docker. Our build script will take longer than necessary. Andrew Lock did an excellent job of writing a solution that makes the restore cacheable, so let's use his solution:
# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly
# copy global files to restore
COPY *.sln *.*config ./
# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
echo "" \
&& echo "-------------" \
&& echo "1/4 RESTORING" \
&& echo "-------------" \
&& echo "" \
&& dotnet restore --verbosity "$VERBOSITY" || exit 1
This method restores the project structure and makes the dotnet restore
cacheable.
Notice how we are using echo
to make our output more readable?
Output of the Dockerfile.
2.2 The build script
Now what should we copy? The base files like the solution and the NuGet config are already in there. Let's copy src
and test
only. This will help us cache builds that are triggered just because some documentation was changed (outside of the directory).
# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test
# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found
RUN echo "" \
&& echo "------------" \
&& echo "2/4 BUILDING" \
&& echo "------------" \
&& echo "" \
&& dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1
2.3 Optional testing and publishing
At the top of our Dockerfile, we've defined the ARG
named EXECUTE_TESTS
. Let's use it to test if we should do any testing. You might want to be able to skip testing to run faster locally.
# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
&& echo "-----------" \
&& echo "3/4 TESTING" \
&& echo "-----------" \
&& echo ""; \
if [ "$EXECUTE_TESTS" = "true" ]; then \
dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
else \
echo "Skipping unit tests"; \
fi; \
echo "" \
&& echo "--------------" \
&& echo "4/4 PUBLISHING" \
&& echo "--------------" \
&& echo "" \
&& dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1
Step 3: runtime stage
The hard part is over: we have our application built. But there are still some choices to make!
3.1 Which runtime base image should we use?
So what runtime image are we going to use? We've built the entire project and checked the size of each resulting image:
Base image | Total output size |
---|---|
mcr.microsoft.com/dotnet/aspnet:7.0 | 218MB |
mcr.microsoft.com/dotnet/nightly/aspnet:7.0-jammy-chiseled | 114MB |
mcr.microsoft.com/dotnet/aspnet:7.0-alpine | 112MB |
Looks like the chiseled project is very promising, as it promises a secure and minimal Debian image, but they have not released it for .NET 7. Most of our projects will run fine on Alpine.
3.2 Go self contained or not?
If you want to go even smaller, you can go to a self contained (and trimmed) application. It is a great way to limit the attack surface even further and a will result in an even smaller image (which is also great). But there are some caveats: your application may take longer to boot and you can't cache your ASP.NET dependencies, so there might be some more network traffic involved. For now we just stick with the old adage as we try to read more on what the best way forward is.
3.3 The final runtime image
We've decided on Alpine for now, so our runtime image looks like this:
#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################
FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime
RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata
ARG PORT API_DLL APP_DIR
# create a new user and change directory ownership
RUN adduser --disabled-password \
--home "$APP_DIR" \
--gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"
# impersonate into the new user
USER dotnetuser
ENV \
ASPNETCORE_URLS=http://*:$PORT \
ASPNETCORE_ENVIRONMENT=Production \
TZ=Europe/Amsterdam \
DOTNET_CLI_TELEMETRY_OPTOUT=1 \
DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
WORKDIR $APP_DIR
COPY --from=build $APP_DIR .
EXPOSE $PORT
ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"
ARG TAG
For security reasons, we create a new user, so we don't run as root.
All together
When we put it all together, we get the following Dockerfile
:
########################
# Configuration settings
########################
ARG API_NAME="Blaze.Prime.Service.Api"
ARG \
API_PROJECT="./src/$API_NAME/$API_NAME.csproj" \
API_DLL="$API_NAME.dll" \
PORT=5000 \
ASPNET_VERSION="7.0" \
# options: q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]
# minimal keeps your pipeline readable while inforing you what's going on
VERBOSITY="minimal" \
APP_DIR="/app" \
TAG="" \
EXECUTE_TESTS="true"
################################################
# Build image, uses ASPNET_VERSION, API_PROJECT,
# VERBOSITY, TAG, APP_DIR, EXECUTE_TESTS
################################################
FROM mcr.microsoft.com/dotnet/sdk:$ASPNET_VERSION AS build
ARG API_PROJECT VERBOSITY APP_DIR
ENV \
TZ=Europe/Amsterdam \
DOTNET_CLI_TELEMETRY_OPTOUT=1 \
DOTNET_NOLOGO=0
WORKDIR /build
# Let's restore the solution, nuget and project files and do a restore
# in cachable layers. This will speed up the build process greatly
# copy global files to restore
COPY *.sln *.*config ./
# copy src files to restore
COPY src/*/*.csproj .
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*}/ && mv $file src/${file%.*}/; done
# copy test files to restore
COPY test/*/*.csproj ./
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*}/ && mv $file test/${file%.*}/; done; \
echo "" \
&& echo "-------------" \
&& echo "1/4 RESTORING" \
&& echo "-------------" \
&& echo "" \
&& dotnet restore --verbosity "$VERBOSITY" || exit 1
# copy dirs that are only needed for building and testing
COPY src ./src
COPY test ./test
# Note on build: don't use --no-restore, sometimes certain packages cannot be
# restored by the dotnet restore. The build will add them, as it has more context (!?)
# example: Package System.Text.Json, version 6.0.0 was not found
RUN echo "" \
&& echo "------------" \
&& echo "2/4 BUILDING" \
&& echo "------------" \
&& echo "" \
&& dotnet build --configuration Release --verbosity "$VERBOSITY" -nowarn:NETSDK1004 || exit 1
# defining the argument here caches the previous layers when the value switches
ARG EXECUTE_TESTS
RUN echo "" \
&& echo "-----------" \
&& echo "3/4 TESTING" \
&& echo "-----------" \
&& echo ""; \
if [ "$EXECUTE_TESTS" = "true" ]; then \
dotnet test --configuration Release --logger "console;verbosity=$VERBOSITY" --no-build || exit 1; \
else \
echo "Skipping unit tests"; \
fi; \
echo "" \
&& echo "--------------" \
&& echo "4/4 PUBLISHING" \
&& echo "--------------" \
&& echo "" \
&& dotnet publish "$API_PROJECT" --configuration Release --output "$APP_DIR" --no-restore -nowarn:NETSDK1004 || exit 1
#################################################
# Runtime image, uses PORT, TAG, API_DLL, APP_DIR
#################################################
FROM mcr.microsoft.com/dotnet/aspnet:$ASPNET_VERSION-alpine as runtime
RUN apk add --no-cache icu-libs krb5-libs libgcc libintl libssl1.1 libstdc++ zlib tzdata
ARG PORT API_DLL APP_DIR
# create a new user and change directory ownership
RUN adduser --disabled-password \
--home "$APP_DIR" \
--gecos '' dotnetuser && chown -R dotnetuser "$APP_DIR"
# impersonate into the new user
USER dotnetuser
ENV \
ASPNETCORE_URLS=http://*:$PORT \
ASPNETCORE_ENVIRONMENT=Production \
TZ=Europe/Amsterdam \
DOTNET_CLI_TELEMETRY_OPTOUT=1 \
DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
WORKDIR $APP_DIR
COPY --from=build $APP_DIR .
EXPOSE $PORT
ENV PROGRAM="$API_DLL"
ENTRYPOINT dotnet "$PROGRAM"
ARG TAG
Pretty cool, right?
Reading list
While studying the subject we found some interesting blogs to read:
- On security: How to build smaller and secure Docker Images for .NET
- On
dotnet restore
& Docker optimizations: Optimising ASP.NET Core apps in Docker - avoiding manually copying csproj files (Part 2) - On the new Chiselled project: Chiselled Ubuntu: the perfect present for your containerised and cloud applications
- On Windows you might get the error "ERROR: failed to solve: error from sender: readdir: open src\*: The filename, directory name, or volume label syntax is incorrect." This is due to a bug in the Docker Moby BuildKit, which cannot handle the wildcards. On PowerShell you can execute
$env:DOCKER_BUILDKIT=0
to disable BuildKit.
Changelog
- 2023-05-26: removed the section on compiling git info into the binary.
- 2023-05-26: installed the
tzdata
package to fix timezone support. - 2023-05-26: notes on how to fix the Windows Docker error: "error from sender: readdir".
- 2023-05-26: nuget.config is now optional.
- 2023-05-26: make sure docker build exits on error.
Top comments (3)
Nicely written and detailed article 🎉 This concepts can be easily applied to any image building process not only for dotnet apps. Kudos!
Thank you for the insight though I will not say I fully grasp it all yet but this has further piqued my interest.
Cool.