Containerize CI pipelines with Earthly

I was a big fan of GitHub Actions to run CI pipelines on our open-source stuff. Compare to other vendors, they did a really good job. It is easy to edit, nicely documented, and has tons of actions to reuse. On the other hand, the Github Actions vendor locks our projects. We ❤️ GitHub, but vendor locking should be a burning problem.

Before I jump into Earthly, I would like to write about the biggest competitor in space: Dagger. Dagger is a new project from Docker Inc. (whatever they called actually), targeting locally reproducible containerized CI pipelines. Sounds fancy, except for one point: it uses CUE language to define jobs. I guess they have the reasons, but I don't want to learn a language to create a build task. Nonsense for me, sorry.

That's where Earthly comes into the picture. It combines Makefile and Dockerfile syntax, so we should jump into with almost zero learning curve 👏. It has a huge amount of features like caching, reusability, live debugging, secret management, etc., for a detailed list please follow the documentation (+100 for documentation).

It's craft time baby!

Let's start with a simple Go lint job and learn Docker In Docker first:

# cat Earthfile
VERSION 0.6

lint:
    FROM earthly/dind:alpine
    WORKDIR /workdir
    COPY . ./
    WITH DOCKER --pull golangci/golangci-lint:v1.49.0
        RUN docker run -w $PWD -v $PWD:$PWD golangci/golangci-lint:v1.49.0 golangci-lint run --timeout 240s
    END

To execute the job we have to install Earthly:

# cat Makefile
BIN_PATH = $(shell pwd)/bin
$(shell mkdir $(BIN_PATH) &>/dev/null)

EARTHLY = $(BIN_PATH)/earthly
earthly:
ifeq (,$(wildcard $(EARTHLY)))
    curl -L https://github.com/earthly/earthly/releases/download/v0.6.23/earthly-linux-amd64 -o $(EARTHLY)
    chmod +x $(EARTHLY)
endif

make earthly && ./bin/earthly --allow-privileged +lint

I think it can't be more simple.

Build and cache dependencies

# cat Earthfile
VERSION 0.6
FROM golang:1.18.0
WORKDIR /workdir

deps:
    COPY go.mod go.sum ./
    RUN go mod tidy
    RUN go mod download
    SAVE ARTIFACT go.mod AS LOCAL go.mod
    SAVE ARTIFACT go.sum AS LOCAL go.sum

test:
    FROM +deps
    COPY . ./
    RUN make _test

I want to write something here, but code describes everything. To spin up job, you have to execute the following command, this time without --allow-privileged:

./bin/earthly +test

Automation, automation, automation, ...

First I would like to execute jobs before I push the change into remote Git. Husky looks like a pretty handy tool to manage Git hooks.

Download Husky:

# cat Makefile
husky:
    GOBIN=$(BIN_PATH) go install github.com/automation-co/husky@v0.2.5

make husky && ./bin/husky init

Edit hook files:

# cat .husky/hooks/pre-commit
#!/bin/bash

husky install

# cat .husky/hooks/pre-push
#!/bin/bash

if [[ "$SKIP_GIT_PUSH_HOOK" ]]; then exit 0; fi

set -e

if git status --short | grep -qv "??"; then
    git stash
    function unstash() {
        git reset --hard
        git stash pop
    }
    trap unstash EXIT
fi

make lint test

git diff --exit-code --quiet || (git status && exit 1)

Install hooks:

./bin/husky install

Create Makefile targets:

# cat Makefile
lint: earthly
    $(EARTHLY) -P +lint

test: earthly
    $(EARTHLY) +test

And GitHub Action integration too, we still need to run jobs at the central place:

name: Lint and test
on: [push]
jobs:
  lint:
    name: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: make lint

  go-test:
    name: go tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: make test

Enjoy 🎉

In this way, before each commit, Husky installs the latest version of hooks and then executes all the tasks before the push. GitHub executes actions after the push, so we did what we could to deliver high-quality software.