Originally published at https://sagikazarmark.hu.
I've been playing with Dagger for months now using it in various projects. In this post, I'll share my experience with using Dagger to build CI pipelines for Go libraries.
TL;DR: Check out this repository for a complete example.
Important: I'm going to focus on Go library specific details and will not explain Dagger basic concepts. Please check out the documentation for an introduction into Dagger.
Go library CI
The easiest way to evaluate Dagger for the Go library use case is to compare it to an existing solution. It's probably fair to say that GitHub Actions dominates the CI market for Open Source Software these days, so it makes sense to compare building a pipeline with Dagger to an existing GitHub Actions workflow.
Here is a simple one for a Go library:
name: CI
on:
push:
branches:
- main
pull_request:
jobs:
test:
name: Test
runs-on: ubuntu-latest
strategy:
matrix:
go: ['1.16', '1.17', '1.18']
steps:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Checkout code
uses: actions/checkout@v3
- name: Test
run: go test -race -coverprofile=coverage.txt -covermode=atomic ./...
- name: Upload coverage
uses: codecov/codecov-action@v3
if: always()
with:
files: coverage.txt
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Set up Go
uses: actions/setup-go@v3
with:
go-version: 1.18
- name: Checkout code
uses: actions/checkout@v3
- name: Lint
uses: golangci/golangci-lint-action@v3
In general terms it consists of the following steps:
- Build matrix to run tests (in parallel) for different environments and settings (eg. multiple Go versions)
- Static analysis and linters
- Publishing analysis results (eg. code coverage) to a third-party service
There are way more complex pipelines out there that run various static analysis tools and integrate with many more services, but most of them fall into one of the three categories above. (Actually, there is a fourth one, but that's hardly ever relevant for a Go library: artifact publishing)
Let's see how we can build a pipeline with Dagger!
Build matrix
Go libraries generally support at least two (or more) Go versions which means CI has to execute tests on all supported versions.
Dagger allows nesting actions, so a naive implementation of a build matrix just repeats the same steps with different Go versions:
test: {
"1.16": go.#Test & {
source: client.filesystem["."].read.contents
package: "./..."
_image: go.#Image & {
version: "1.16"
}
input: _image.output
}
"1.17": // go.#Test ...
"1.18": // go.#Test ...
}
The different actions then can be executed either per version (in parallel CI jobs) or at the same time (eg. locally):
dagger do test 1.18 # Run tests for a single Go version
dagger do test # Run tests for all Go versions
Naturally, this isn't the most optimal solution: it doesn't work well with multiple dimensions and it requires a lot of duplication even for a single dimension.
An alternative solution is using templating in CUE. It reduces the amount of code and works well with multiple dimensions:
test: {
"1.16": _
"1.17": _
"1.18": _
[v=string]: go.#Test & {
source: client.filesystem["."].read.contents
package: "./..."
_image: go.#Image & {
version: v
}
input: _image.output
}
}
Admittedly, this is not a real build matrix either as all variations have to be listed manually, but for most of the use cases it's close enough.
Static analysis
The Go ecosystem has an exceptionally large number of static analysis tools and chances are not all of them are going to be available in Dagger Universe (Dagger's central repository of actions).
Writing a custom action allows integrating any arbitrary tool into a Dagger plan.
First, make sure that cue.mod/module.cue contains a module name:
module: "github.com/sagikazarmark/dagger-go-library"
Then create a new CUE file for the tool definition (eg. ci/golangci/lint.cue):
// Lint using golangci-lint
#Lint: {
// Source code
source: dagger.#FS
// golangci-lint version
version: *"1.46" | string
// Timeout
timeout: *"5m" | string
_image: docker.#Pull & {
source: "index.docker.io/golangci/golangci-lint:v\(version)"
}
go.#Container & {
name: "golangci_lint"
"source": source
input: _image.output
command: {
name: "golangci-lint"
flags: {
run: true
"-v": true
"--timeout": timeout
}
}
}
}
Generally speaking, an action needs a source code input and some tool specific parameters as inputs.
The action can then be added to the plan:
lint: {
"go": golangci.#Lint & {
source: client.filesystem["."].read.contents
version: "1.46"
}
}
Note: GolangCI has been recently added to the alpha channel of Dagger Universe, so you can use that one instead.
Publishing analysis results
It's common for libraries to publish (static) analysis results after test runs to analyze and visualize various code metrics.
One of those metrics is code coverage that tells us how much of the source code was actually executed (covered) by test runs. Code coverage information is generated by the go test tool, so the output of the test step needs to be passed to another step.
Dagger handles these kind of dependencies well, all we need to do is reference the output of one action in another:
test: {
"1.16": _
"1.17": _
"1.18": _
[v=string]: {
_test: go.#Test & {
// ...
command: flags: {
"-covermode": "atomic"
"-coverprofile": "/coverage.out"
}
export: files: "/coverage.out": _
}
_coverage: codecov.#Upload & {
// Merge the coverage file with the source code (for VCS information)
_write: core.#WriteFile & {
input: client.filesystem["."].read.contents
path: "/coverage.out"
contents: _test.export.files."/coverage.out"
}
source: _write.output
file: "coverage.out"
}
}
}
Note: You can find the definition for the Upload action here.
Running it on GitHub Actions
Although we have a plan, it still needs to be executed somewhere. GitHub Actions is a perfect candidate, especially for Open Source Go libraries.
Rewriting the above workflow, we can easily run Dagger actions:
name: Dagger
on:
push:
branches:
- main
pull_request:
jobs:
test:
name: Test
runs-on: ubuntu-latest
strategy:
matrix:
go: ['1.16', '1.17', '1.18']
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Dagger
uses: dagger/dagger-for-github@v3
with:
cmds: |
project update
do check test go ${{ matrix.go }}
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Dagger
uses: dagger/dagger-for-github@v3
with:
cmds: |
project update
do check lint
There is one more change we have to make to our plan: Some actions (like coverage upload) might need detailed VCS information. One option is to pass them manually to each action or let the tools collect them from the environment (eg. environment variables):
client: env: {
CI: string | *""
GITHUB_ACTIONS: string | *""
GITHUB_ACTION: string | *""
GITHUB_HEAD_REF: string | *""
GITHUB_REF: string | *""
GITHUB_REPOSITORY: string | *""
GITHUB_RUN_ID: string | *""
GITHUB_SERVER_URL: string | *""
GITHUB_SHA: string | *""
GITHUB_WORKFLOW: string | *""
}
actions: {
test: {
"1.16": _
"1.17": _
"1.18": _
[v=string]: {
_test: go.#Test & {
// ...
}
_coverage: codecov.#Upload & {
// ...
// No need to run coverage upload unless running in CI
dryRun: client.env.CI != "true"
env: {
GITHUB_ACTIONS: client.env.GITHUB_ACTIONS
GITHUB_ACTION: client.env.GITHUB_ACTION
GITHUB_HEAD_REF: client.env.GITHUB_HEAD_REF
GITHUB_REF: client.env.GITHUB_REF
GITHUB_REPOSITORY: client.env.GITHUB_REPOSITORY
GITHUB_RUN_ID: client.env.GITHUB_RUN_ID
GITHUB_SERVER_URL: client.env.GITHUB_SERVER_URL
GITHUB_SHA: client.env.GITHUB_SHA
GITHUB_WORKFLOW: client.env.GITHUB_WORKFLOW
}
}
}
}
}
Check out the example repository for more details and examples (like caching on GitHub Actions).
Conclusion
The first question that needs to be answered: Is it really worth it?
Well, it depends.
On one hand, building pipelines locally is certainly easier
than doing so on a CI server. It's also comforting to know that if it runs locally, it'll run on the CI server.
On the other hand, CI pipelines for Go libraries tend to be simple and with GitHub Actions the building blocks are as reusable as Dagger actions are. Native CI runs also tend to be a bit faster due to the lack of extra runtime initialization, but that performance hit can be minimized with caching and is less noticeable in large projects.
So it really comes down to personal preference.
My personal favorite feature in Dagger is its portability: I like that I can run and build it on my own machine, but portability really becomes important for complex pipelines and large projects. For a simple Go library, I'll probably stick to Makefiles and GitHub Actions.
Make sure to check out the complete example for more details.
Top comments (0)