We've been using containers for software packaging for a long time. But let’s face it, Docker's user experience isn't really the greatest. Building and testing usually entails having a bunch of Makefiles, fiddly Dockerfiles, a bit of black magic, and a bundle of Bash scripts. It’s a path that can lead into insanity.
And it’s a shame because Docker could be much friendlier. Earthly developers certainly believe in this: they have created a tool that fills the space between Docker and Make.
At the end of this tutorial, you’ll know how Earthly works. As a bonus, you'll have a pipeline that builds, tests, and pushes a Docker image into a remote repository.
How Earthly works
Earthly is a Docker-based build tool. It does not, however, replace language-specific tools like Maven, Gradle, or Webpack. Instead, it leverages and integrates with them — acting as a glue.
Using Docker containers as the core mechanic to achieve repeatability, every Earthly command runs in an isolated environment and is effortlessly parallelized whenever possible.
Earthly consists of the CLI binary and a Docker image called earthly/buildkit, based on Docker’s BuildKitd.
Earthly wants you to get the same experience everywhere. BuildKit’s role is to yield portable Docker-compatible layers. Thus, we get the same result whether on our local machine and in our CI systems.
Does Earthly deliver on its promises?
Earthly promises better, repeatable builds. But, does it deliver? Here are some of the best features and some of the problems found while doing a test drive.
Pros:
- Earthly, like Docker, is language-agnostic. Its syntax feels familiar since it mixes Make and Dockerfiles.
- Their site has good documentation.
- Configuration is low maintenance. There are only a few settings to tweak.
- Useful even in the cases you don’t need Docker. Container images are only one of the possible outputs. For instance, you can use Earthly to test and compile binaries in a clean environment.
- The import system works well in multirepo and monorepos.
Cons:
- It doesn’t actually replace Makefile or Docker Compose. And you will still need some shell scripts.
- It’s slower than native Docker because it relies on copying files instead of mounts. Of course, this is by design to ensure repeatability.
- Multiplatform builds sometimes use emulation, so it may be too slow for some use cases.
- Error messages can get really confusing.
Setting up Earthly
Earthly configuration files are called, as typical in software engineering, Earthfiles
. Opening one reminds us of Dockerfiles with a twist of Makefiles.
Earthly takes the syntax from Docker and expands it for a general use case. While Dockerfiles are only meant to produce container images, Earthly can generate artifacts.
The following Earthfile was taken from Earthly’s official guide. It describes two targets: build
and docker
.
FROM golang:1.15-alpine3.13
WORKDIR /go-example
build:
COPY main.go .
RUN go build -o build/go-example main.go
SAVE ARTIFACT build/go-example /go-example AS LOCAL build/go-example
docker:
COPY +build/go-example .
ENTRYPOINT ["/go-example/go-example"]
SAVE IMAGE go-example:latest
The build
target compiles a Go program inside a container, while docker
copies it into a production-ready image that can be pushed into any container registry and run anywhere.
The FROM
keyword works as you would expect if familiar with Dockerfiles. Likewise, COPY
, ENTRYPOINT
, WORKDIR
, and RUN
, behave similarly. Two concepts deserve a pause to explain further:
- targets: the thing we want to build. Earthly uses a target-based system inspired by Make.
- references: targets can reference other targets. As in Make, Earthly will sort dependencies and do the right thing.
- extended commands: Earthly features specialized commands such as SAVE IMAGE and SAVE ARTIFACT.
To run this Earthfile we need to run: earthly +TARGET
. The first time, Earthly will pull the image and create a cache volume. The cache will persist the build files between runs.
$ earthly +build
buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
golang:1.15-alpine3.13 | --> Load metadata linux/arm64
context | --> local context .
+base | --> FROM golang:1.15-alpine3.13
context | transferred 1 file(s) for context . (2.1 MB, 4 file/dir stats)
+base | *cached* --> WORKDIR /go-example
+build | *cached* --> COPY main.go .
+build | *cached* --> RUN go build -o build/go-example main.go
output | --> exporting outputs
================================ SUCCESS [main] ================================
+build | Artifact +build/go-example as local build/go-example
Or we can directly go to the Docker generation target with:
$ earthly +docker
+base | --> FROM golang:1.15-alpine3.13
+base | *cached* --> WORKDIR /go-example
context | transferred 3 file(s) for context . (2.1 MB, 4 file/dir stats)
+build | *cached* --> COPY main.go .
+build | *cached* --> RUN go build -o build/go-example main.go
+build | *cached* --> SAVE ARTIFACT build/go-example +build/go-example AS LOCAL build/go-example
+docker | --> COPY +build/go-example ./
output | [██████████] exporting layers ... 100%
output | [ ] exporting manifest
================================ SUCCESS [main] ================================
+docker | Image +docker as go-example:latest
+build | Artifact +build/go-example as local build/go-example
Since the docker
target depends on build
, Earthly first runs the build step and then proceeds to build and export the image into the local Docker directory.
Extending Earthfiles
Let’s try Earthly in a more involved example. Please go ahead and fork this demo repository.
The provided Dockerfile builds a Node Alpine container image.
FROM node:14.17-alpine3.12
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm install
COPY src src
EXPOSE 3000
ENTRYPOINT ["npm", "start"]
You can build it with: docker build . -t mydemo
and then run it as: docker run -it -p 3000:3000 mydemo
. Browsing the localhost on port 3000 should print a "hello, world" message.
How would we go about turning this into a working Earthfile? Let’s start one from scratch.
The first two lines remain the same and will be used for every target.
FROM node:14.17-alpine3.12
WORKDIR /app
Now we add the build target, which downloads dependencies and copies the source.
build:
COPY package.json package-lock.json ./
COPY --if-exists node_modules node_modules
RUN npm install
COPY src src
We have added a COPY
statement to copy the node_modules folder into the build environment.
Since npm runs in the container, we need to copy its output files back outside. We do this with Earthly’s special command SAVE ARTIFACT ... AS LOCAL
.
SAVE ARTIFACT node_modules AS LOCAL ./node_modules
SAVE ARTIFACT package.json AS LOCAL ./package.json
SAVE ARTIFACT package-lock.json AS LOCAL ./package-lock.json
Running earthly +build
should install the Node dependencies. The end result of this process is similar to what we would get if we run npm install
directly, with the added benefit that we get the same outcome everywhere.
Next, we should create the Docker image. We can achieve this with:
docker:
FROM +build
EXPOSE 3000
ENTRYPOINT ["npm", "start"]
SAVE IMAGE semaphore-demo-earthly:latest
The new target picks up from where the build stopped. It adds the start command and exports the image into the host’s local Docker registry.
Testing with Earthly
One of the most powerful features Earthly brings is the ability to extend Dockerfiles with tests. Let’s add unit tests by extending build
with a target that runs: npm test
.
tests:
FROM +build
COPY spec spec
RUN npm test
Now we can run earthly +tests
to get the results.
Adding a linting target is also trivial:
lint:
FROM +build
COPY .jshintrc ./
RUN npm run lint
To run more involved tests, such as integration tests or end-to-end tests, we can reuse existing Docker Compose manifests in order to bring up other containers during the integration stage.
integration-tests:
FROM +build
COPY docker-compose.yml ./
COPY integration-tests integration-tests
WITH DOCKER --compose docker-compose.yml --service db
RUN sleep 10 && npm run integration-tests
END
Earthly uses a Docker-in-Docker approach to start helper containers:
- Copies
docker-compose.yml
and the integration tests in the main container (the application container). - Starts a PostgreSQL container inside the main container. The special
WITH DOCKER
command accepts precisely oneRUN
statement. - Runs the integration tests.
Our Earthfile is good enough for now. Let’s upload it into the repository so we can configure a CI/CD pipeline in the next section.
$ git add Earthfile
$ git commit -m "push Earthfile"
$ git push origin master
Earthly Continuous Integration Pipeline
We are now all set to test the application with Earthly and continuous integration. Our pipeline will pull to and push from Docker Hub, something that will require authentication with the service.
Authentication in Semaphore happens via secrets. To create an encrypted secret, go to your organization menu and click on Settings > Secrets. Then, press Create secret and type the variables DOCKER_USERNAME
and DOCKER_PASSWORD
as shown below:
Next, initialize the forked demo repository. The procedure for adding a project is detailed in the getting started guide. Go check it out if this is the first time using Semaphore.
Since Earthly doesn’t come preinstalled in the Ubuntu CI image, we need to install it for every job. The simplest way of achieving this is with a pipeline-level prologue. The commands in the global prologue are executed before every job in the pipeline.
The Earthly installation command for Linux is:
sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'
Click on the pipeline and type the line in the prologue section.
Build block
The first block will start a build stage. In it, we will download and cache Node dependencies.
Type the following commands in the job:
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci +build
cache store
We’ll run these commands or a variation of them in every successive job in the pipeline:
- docker login: logs into Docker Hub with the credentials defined in the secret.
- checkout: clones the repository into the CI machine.
-
cache: With the cache command, we can store
node_modules
in the project-level storage provided by Semaphore. - earthly ci: runs Earthly with some CI-optimized settings.
To finish the block, enable the “dockerhub” secret, so the variable is decrypted and exported in the job.
Tests block
The next block will run fast tests such as unit tests and linting. Add the following commands in the block’s prologue:
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
Next, create two jobs:
earthly --ci +tests
And:
earthly --ci +lint
Integration tests block
The last job will run the integration test. Type the following commands. The -P
switch is needed to enable privileged mode inside Docker (needed for Docker-in-Docker).
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly --ci -P +integration-tests
Don’t forget to check that the secret is enabled in all blocks. When ready, run the workflow to try the CI pipeline.
Continuous Delivery with Earthly
The final result of a CI/CD pipeline is to deliver something that can be deployed. Here’s where the continuous delivery part comes into play. We’ll push the image to a remote repository so it can be later deployed in production.
A Semaphore workflow can span multiple pipelines, as long as they’re connected by promotions. Go back to the workflow editor and click on Add Promotion. Click the new pipeline and copy the global Earthly install commands, again:
sudo /bin/sh -c 'wget https://github.com/earthly/earthly/releases/latest/download/earthly-linux-amd64 -O /usr/local/bin/earthly && chmod +x /usr/local/bin/earthly && /usr/local/bin/earthly bootstrap --with-autocomplete'
The block in the pipeline will contain a single job to generate and push the Docker image into Docker Hub. The commands are as follows:
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
cache restore
earthly +docker
docker tag semaphore-demo-earthly $DOCKER_USERNAME/semaphore-demo-earthly
docker push $DOCKER_USERNAME/semaphore-demo-earthly
We use earthly +docker
to export the image into the CI machine, then a combination of docker tag
and docker push
to send it to the remote registry.
The only thing left is to enable the Docker Hub secret … and we’re done!
Rerun the workflow and click on promote to try pushing the image.
What do you think?
The Earthly project is young and under heavy development, with many features still in the experimental stage (we stuck to stable features during the course of the post). It’s not perfect, but it feels like a massive step in the right direction.
Skill up your Docker knowledge with these tutorials:
- Kubernetes vs. Docker: Understanding Containers in 2021
- How To Deploy a Go Web Application with Docker
- Dockerizing a Node.js Web Application
- Dockerizing a Ruby on Rails Application
Have you tried Earthly? Tell us about your experience.
Top comments (0)