TECH SCHOOL

Posted on Sep 25, 2020 • Edited on Jun 26, 2021

How to setup Github Actions for Go + Postgres to run automated tests

#go #github #devops #tutorial

Continuous integration (CI) is one important part of the software development process where a shared code repository is continuously changing due to new work of a team member being integrated into it.

To ensure the high quality of the code and reduce potential errors, each integration is usually verified by an automated build and test process.

In this article, we will learn how to setup that process using Github Action to automatically build and run unit tests for our simple bank project, which is written in Golang and uses PostgreSQL as its main database.

Here's:

Link to the full series playlist on Youtube
And its Github repository

How Github Actions works

Github Action is a service offered by Github that has similar functionality as other CI tools like Jenkins, Travis, or CircleCI.

Workflow

In order to use Github Actions, we must define a workflow. Workflow is basically an automated procedure that’s made up of one or more jobs. It can be triggered by 3 different ways:

By an event that happens on the Github repository
By setting a repetitive schedule
Or manually clicking on the run workflow button on the repository UI.

To create a workflow, we just need to add a .yml file to the .github/workflows folder in our repository. For example, this is a simple workflow file ci.yml:

name: build-and-test

on:
  push:
    branches: [ master ]
  schedule:
    - cron:  '*/15 * * * *'

jobs:
  build:
    runs-on: ubuntu-latest

The name of this workflow is build-and-test. We can define how it will be triggered using the on keyword.

In this flow, there's an event that will trigger the workflow whenever a change is pushed to the master branch, and another scheduled trigger that will run the workflow every 15 minute.

Then we define the list of jobs to run in the jobs section of the workflow yaml file.

Runner

In order to run the jobs, we must specify a runner for each of them. A runner is simply a server that listens for available jobs, and it will run only 1 job at a time.

We can use Github hosted runner directly, or specify our own self-hosted runner.

The runners will run the jobs, then report the their progress, logs, and results back to Github, so we can easily check it on the UI of the repository.

We use the run-on keyword to specify the runner we want to use.

jobs:
  build:
    runs-on: ubuntu-latest

In this example workflow, we’re using Github’s hosted runner for Ubuntu’s latest version.

Job

Now let’s talk about Job. A job is a set of steps that will be executed on the same runner.

Normally all jobs in the workflow run in parallel, except when you have some jobs that depend on each other, then they will be run serially.

The jobs are listed inside the workflow under the jobs keyword.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Build server
        run: ./build_server.sh
  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - run: ./test_server.sh

In this example, we have 2 jobs:

The first one is build, which has 2 steps: check out code, and build server.
The second job is test, which will run the tests of the application.

Here we use the needs keyword to say that the test job depends on the build job, so that it can only be run after our application is successfully built.

This test job only has 1 step that runs the test_server.sh script.

Step

Steps are individual tasks that run serially, one after another within a job. A step can contain 1 or multiple actions.

Action is basically a standalone command like the one that run the test_server.sh script that we’ve seen before. If a step contains multiple actions, they will be run serially.

An interesting thing about action is that it can be reused. So if someone has already written a github action that we need, we can actually use it in our workflow.

Let’s take a look at this example.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v2
      - name: Build server
        run: ./build_server.sh

Here we use the steps keyword to list out all steps we want to run in our job.

The first step is to check out the code from Github to our runner machine. To do that, we just use the Github actions checkout@v2, which has already been written by the Github action team.

The second step is to build our application server. In this case, we provide our own action, which is simply running the build_server.sh script that we’ve created in the repository.

And that’s it!

Summary

Before jumping in to coding, let’s do a quick summary:

We can trigger a workflow by 3 ways: event, scheduled, or manually.
A workflow consists of one or multiple jobs.
A job is composed of multiple steps.
Each step can have 1 or more actions.
All jobs inside a workflow normally run in parallel, unless they depend on each other, then in that case, they run serially.
Each job will be run separately by a specific runner.
The runners will report progress, logs, and results of the jobs back to github. And we can check them directly on Github repository’s UI.

Setup a workflow for Golang and Postgres

Alright, now let’s learn how to setup a real workflow for our Golang application so that it can connect to Postgres, and run all the unit tests that we’ve written in previous lectures whenever new changes are pushed to Github.

Use a template workflow

In our simple bank repository on Github, let’s select the Actions tab.

Github knows that our project is written mainly in Go, so it suggests us to setup the workflow for Go. Let’s click this setup button.

As you can see, a new file go.yml is being created under the folder .github/workflows of our repository with this template:

name: Go

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  build:
    name: Build
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Get dependencies
      run: |
        go get -v -t -d ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi

    - name: Build
      run: go build -v .

    - name: Test
      run: go test -v .

We can edit this file directly here using this Github editor. However, I prefer to add the file to our local repository first, then edit it locally with visual studio code before pushing to Github.

Create a workflow yaml file

So let’s open our simple bank project folder in the terminal. I’m gonna create a new folder .github/workflows.

❯ cd ~/Projects/techschool/simplebank
❯ mkdir -p .github/workflows

Then create a new Yaml file for our workflow inside this folder. You can name it whatever you want, just make sure it has yml extension. For me, I’m just gonna use ci.yml to be simple.

❯ touch .github/workflows/ci.yml

Now let’s open this project in visual studio code.

Here we can see the ci.yml file under .github/workflows folder. Let’s go back to Github and copy the go.yml file content, then paste it to our ci.yml file.

First we need to set a name for this workflow, for example: ci-test. This name will be displayed in our Github repository’s Actions page.

name: ci-test

Config trigger events

Then we define the events that can trigger this workflow. Normally we would want to run tests whenever there’s a change being pushed to the master branch, or when there’s a pull request to merge into the master branch.

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

There are many other events that you can use. Please refer to the Github Actions documentation to know more about them.

Setup jobs

Next we’re gonna setup the jobs. In this template that Github provides us, we have only 1 job.

Its name is build, and it runs on a Ubuntu runner. I think we should rename this job to test because that’s the main purpose of it.

jobs:

  test:
    name: Test
    runs-on: ubuntu-latest

There are several steps in this job.

Step 1: Install Go

The first step is to setup or install Go into the runner. In this step, we just need to use the existing Github action called setup-go@v2.
```
    steps:

    - name: Set up Go 1.x
    uses: actions/setup-go@v2
    with:
        go-version: ^1.15
    id: go
```
We use the with keyword to provide input parameters to this action. In this case, we can ask it to use a specific version of Go, such as version 1.15.

The id field is just a unique identifier of this step. We might need it if we want to refer to this step in other context.
Step 2: Checkout code

The second step is to check out the code of this repository into the runner. To do that, we also reuse an existing action: checkout@v2.
```
    - name: Check out code into the Go module directory
    uses: actions/checkout@v2
```
Step 3: Get dependencies

The next step is to get all the dependencies, or external packages that our project is using.
```
    - name: Get dependencies
    run: |
        go get -v -t -d ./...
        if [ -f Gopkg.toml ]; then
            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
            dep ensure
        fi
```
In fact, we don’t need this step because go mod will automatically download missing libraries when we build the application or run the tests. So let's remove it!

The build step is also not necessary because the application will be built automatically when we run go test.
```
    - name: Build
    run: go build -v .
```
Step 4: Run the tests

So the last remaining step is to run our unit tests. We already have a make test command defined in the Makefile for this purpose. Therefore, all we have to do in this step is to call it:
```
    - name: Test
    run: make test
```

Push the workflow to Github

So we’re done with the first basic version of our CI workflow:

name: ci-test

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  test:
    name: Test
    runs-on: ubuntu-latest

    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.15
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Test
      run: make test

It might not work yet because we haven’t setup the Postgres database. But let’s just push it to Github to see how it run:

❯ git status
❯ git add .
❯ git commit -m "init CI workflow"

Here we first run git status to check the status of our local repository. After that, we run git add . to add all new changes to the list of our commit.

Then run git commit with a message saying init CI workflow to commit it to our local repository. And finally, we run git push origin master to push this change to our remote repository on Github.

Now let's go back to our Github repository page and select Actions tab.

Now we can see our ci-test workflow here, and there’s a new run of it for our commit. When we open this run, we can see 1 job in progress: Test.

All steps are listed on the right. The Setup job, Setup Go, and Checkout code steps are finished successfully because there’s a green tick in front of them. The Test step is still running because there’s a yellow circle before it.

Now the Test step has finished, but it failed. We know that because of the red x icon next to it.

This is expected, because as we’re seeing in the logs, the code cannot connect to port 5432 of Postgres, since we haven’t set it up in our workflow yet. So let’s do that now!

Add Postgres service

Let’s search for github action postgres, and open this official Github Action documentation page about creating Postgres service containers.

Here in this section, we can see that Postgres is declared as an external service of this job. Let’s copy this block of code and paste it to our workflow file.

So we use the services keyword to specify a list of external services that we want to run together with our job. In this case, we only need 1 service, which is Postgres.

And since we’re using Postgres version 12 in our project, let’s set this docker image name to postgres:12. You can check out available versions and tags of this Postgres image on Docker Hub.

    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

Next we need to set some environment variables for the credentials to access the database.

If you still remember, we’re using user = "root", password = "secret", and database = "simple_bank" in our local Postgres container. So let’s set the same value here for our CI workflow.

The health check option is very important because it is used by the runner to check if Postgres has started successfully or not, so that it can know when to run the next steps in the workflow.

That’s great because we only want our tests to be run after Postgres is started. Otherwise, the tests will still fail because it cannot connect to the database, right?

Add run migrations step

OK, now the Postgres service is defined, but in order for our tests to run successfully, we also need to run db migrations to create the correct database schema for our application.

So let’s define a new step here, after the check out code step. Its name will be Run migrations. And the only action it needs to do is to run make migrateup.

    - name: Run migrations
      run: make migrateup

Alright, now let’s try to push this new workflow changes to Github to see what will happen.

OK, now in our repository’s Actions page, we can see a new run for our new commit. Here the Test job is still running. The job is set up successfully, and now it’s initializing the containers.

From the logs, we know that it’s still waiting for Postgres service to be ready. As soon as Postgres is up, all following steps are run immediately.

Here we can see a log saying Postgres service is healthy. The Setup Go step is also successful. Then it checkout new the code.

Now the migrations step is failing because migrate is not found. We forgot to install the golang-migrate CLI tool to run the migrations.

Install golang-migrate CLI

So let’s search for golang migrate, and open this Github page documentation.

There are several options depending on the OS that you use. We’re using Ubuntu for our runner, so I’m gonna copy this curl command to download a pre-built binary of the migrate CLI.

❯ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz

Now in the workflow, let’s add a new Step to Install golang migrate. Then in the run action, let’s paste in the curl command.

We have to set the correct URL for the version of migrate CLI and the platform that we want to use. So let’s click on this Release downloads link.

The latest release is version 4.12.2. And since our ubuntu runner is a linux platform, let’s copy the migrate.linux-amd64.tar.gz link address, then paste it to our curl command.

    - name: Install golang-migrate
      run: curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz

It will download the zip file, and unzip it to give us the migrate binary named migrate.linux-amd64. Now in order for the migrate command to work, we have to move that binary to the /usr/bin folder.

So this step will include more than just 1 curl command. We use this vertical pipe | character here to specify a multi-line commands. Let's add this move command to the step:

    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/
        which migrate

Note that only a superuser can change the content of the /usr/bin folder, so we have to run this command with sudo.

We also add 1 more command: which migrate, just to check if the migrate CLI binary is successfully installed and ready to be used in the runner or not.

Now let's commit the new change of our workflow and push it to Github. Then check our repository’s Actions page.

Now the job is still failing, but this time it fails at the Install golang-migrate step.

From the logs, we can say that the binary file was successfully downloaded. So it might fail because of the move command, or the which migrate command.

OK I know why! That’s because we’re just moving the file migrate.linux-amd64 to /usr/bin, but we don’t rename it to migrate. So when we run which migrate, it cannot find any binary with that name.

All we have to do now is to add migrate to the end of the move command, so that the binary file is moved to /usr/bin with a new name: migrate.

    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/migrate
        which migrate

This will ensure that when the make migrateup command is run, the correct migrate CLI binary will be used.

Alright, let’s add this new change, commit it, and push it to Github. Then go back to our Repository’s Action page to check the job’s status.

It's still failing. However, this time, the Install golang-migrate step is successful. The step that fails is Run migrations.

And the reason is: it still cannot connect to port 5432 of our Postgres container. Why? We’ve already added Postgres to the services list right?

Well, yes! But we haven’t exposed its local port to the external host yet. That’s why our code still cannot connect to the port.

Add port mapping to Postgres service

We can use the ports keyword to specify the ports that we want to expose to the external host, just like what we normally do in our docker compose file. Let’s add it to our CI workflow.

    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

The default port 5432 of Postgres is now available to our job runner to access.

Let’s add this new change, commit it, and push the change to Github. Hopefully this time it will work.

It’s done! All green ticks. So finally our CI-test workflow runs successfully.

After all steps in our workflow are completed, Github do some clean up steps and stop the containers.

And that’s it! We have learned about continuous integration by writing our first Github Action workflow to run Golang unit tests that need to connect to an external Postgres service.

Here's the complete workflow file .github/workflows/ci.yml:

name: ci-test

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:

  test:
    name: Test
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: secret
          POSTGRES_DB: simple_bank
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:

    - name: Set up Go 1.x
      uses: actions/setup-go@v2
      with:
        go-version: ^1.15
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v2

    - name: Install golang-migrate
      run: |
        curl -L https://github.com/golang-migrate/migrate/releases/download/v4.12.2/migrate.linux-amd64.tar.gz | tar xvz
        sudo mv migrate.linux-amd64 /usr/bin/migrate
        which migrate

    - name: Run migrations
      run: make migrateup

    - name: Test
      run: make test

There are a lot of more things that Github Action can do. I encourage you to check out its official documentation to learn more about them.

And that brings us to the end of this article. Thanks a lot for reading, and I will see you guys in the next lecture!

If you like the article, please subscribe to our Youtube channel and follow us on Twitter for more tutorials in the future.

If you want to join me on my current amazing team at Voodoo, check out our job openings here. Remote or onsite in Paris/Amsterdam/London/Berlin/Barcelona with visa sponsorship.

DEV Community