CockroachDB Integration Testing

Testing is vitally important to the success of software and software that connects to databases is no exception. There are a number of ways to test database code, ranging from mocks, to full integration tests that talk to an actual database.

When writing mocks, I've had some great experiences with Data Dog's sqlmock and I'd recommend that you check it out. In this post however, I'm going to devle into the steps involved in integration testing a CockroachDB database using docker-compose.

All code can be found in GitHub.

Test code

Firstly, let's go over the test code. It's super contrived but hopefully gets the point across. If anything's not clear, I'd be happy to discuss, send me a message on Twitter.

I use the TestMain method to bootstrap my tests, which allows me to capture command line arguments and gives my tests a logical place to setup and teardown database connections.

This post gives a lovely introduction to using Go's flags with your tests and I'll be using lessons from there to determine whether or not to run database integration tests in this example.

My simple test code uses two global variables (the horror!) and depending on your test directory structure, you might want to do this differently. One variable holds the flag value that will determine whether database integration tests are run and the other holds a reference to the database connection itself.

var (
    dbTests *bool
    db      *sql.DB
)

TestMain is an entry point for Go tests. In the following code you'll see that I'm capturing whether the caller wants to run database integration tests and setting up/tearing down the database connection if they've asked for database integration tests to be run.

func TestMain(m *testing.M) {
    dbTests = flag.Bool("db", false, "run database integration tests")
    flag.Parse()

    if *dbTests {
        setupDatabase()
    }

    exitCode := m.Run()

    if *dbTests {
        teardownDatabase()
    }

    os.Exit(exitCode)
}

The database setup code expects an environment variable called "CONN_STR" and crashes the test if it doesn't exist. Hence I wrap calls to setupDatabase in a check to *dbTests, allowing me to run unit tests that don't care about database tests (or connection strings from the environment) without having to provide this information.

The guts of setupDatabase is the database connectivity logic. A call is made to open a connection to the database and another is made to verify that the connection is open and ready:

func setupDatabase() {
    connStr, ok := os.LookupEnv("CONN_STR")
    if !ok {
        log.Fatal("connection string env var not found")
    }

    var err error
    if db, err = sql.Open("postgres", connStr); err != nil {
        log.Fatalf("error connecting to the database: %v", err)
    }

    if err = db.Ping(); err != nil {
        log.Fatalf("error pinging the database: %v", err)
    }
}

The teardownDatabase function could simply be replaced with a call to db.Close() but I preferred the consistency in the TestMain function, so decided to keep it:

func teardownDatabase() {
    db.Close()
}

Finally, I've written an integration test. It currently does nothing useful, further to logging what it's doing (running database integration tests or not):

func TestStuff(t *testing.T) {
    if !*dbTests {
        t.Skipf("not running database tests")
    }

    t.Log("write some database tests...")
}

Containerise

To be able to run our database integration tests reliably and get repeatable results, we're going to need to run 2 Docker containers, one to host an instance of CockroachDB and another to execute our tests.

Dockerfile

To start with, let's create a Dockerfile to run our tests. The below spins up a very lightweight Alpine-based container, copies our project's top-level directory into an "app" directory inside the container, and sets the working directory for the container to that directory:

FROM alpine:latest
COPY . /app
WORKDIR /app

Build

As we're using the Alpine base image, we won't have the means to run Go code (or Go tests). We'll therefore build the tests so they can simply be executed within the container, just like any other executable.

The following builds a test executable called "crdb-test" for the Alpine Linux container that when executed, will be the same as running go test ./... -v, without having to rely on the Go executable:

$ GOOS=linux go test ./... -v -c -o crdb-test

Image

Next, we need to build the Docker image. The following builds a docker image called "crdb-test", using the Dockerfile we created earlier:

$ docker build -t crdb-test .
Sending build context to Docker daemon   6.18MB
Step 1/3 : FROM alpine:latest
 ---> 5cb3aa00f899
Step 2/3 : COPY . /app
 ---> Using cache
 ---> 936e88998fa5
Step 3/3 : WORKDIR /app
 ---> Using cache
 ---> 30e76628fce3
Successfully built 30e76628fce3
Successfully tagged crdb-test:latest

At this point, you can run the Docker image and hop into it via the terminal. The following command will open up a terminal and put you straight into the container's working directory, allowing you to poke around and check that your "crdb-test" test binary is where you're expecting it to be:

$ docker run -it crdb-test:latest sh

Compose

To tie our test container and the CockroachDB container together, we'll use docker-compose. Here's the full file, which I'll break into smaller parts and describe below:

version: '3'

services:
  cockroach_test:
    image: cockroachdb/cockroach:v2.1.6
    command: start --insecure
    ports:
      - "26257:26257"
      - "8080:8080"
    volumes:
      - ./cockroach-data/roach1:/cockroach/cockroach-data
    logging:
      driver: "none"
    networks:
    - app-network

  app_test:
    image: crdb-test:latest
    environment:
      CONN_STR: postgresql://root@cockroach_test:26257/defaultdb?sslmode=disable
    command: ./crdb-test -db
    working_dir: /app
    depends_on:
    - cockroach_test
    networks:
    - app-network

networks:
  app-network:
    driver: bridge

I've named the first container (or "service") "cockroach_test". This will become the DNS name that I'll use to connect my test container to.

cockroach_test:
  image: cockroachdb/cockroach:v19.2.0
  command: start --insecure
  logging:
    driver: "none"
  networks:
  - app-network

For the image parameter, I pass the current latest version of the CockroachDB Docker image. You can substitute another version of the database for your tests.

For the command parameter, I pass start --insecure. These are the command line arguments that you'd pass to the cockroach CLI if running outside of Docker.

For the logging parameter, I pass an argument that disables logging for the CockroachDB container. This is entirely optional and just means the logging output is more test-focused. Helpful if you're trawling through test logs to figure out why a test is unexpectedly failing!

I've also configured a bridge network for the continers to talk to each other, so for the networks parameter, I pass "app-network", the name of that network.

I spin up a container to execute my test code and refer to it as "app_test".

app_test:
  image: crdb-test:latest
  environment:
    CONN_STR: postgresql://root@cockroach_test:26257/defaultdb?sslmode=disable
  command: ./crdb-test -db
  working_dir: /app
  depends_on:
  - cockroach_test
  networks:
  - app-network

For the image parameter, I refer back to the image I built earlier, "crdb-test", suffixing ":latest", the default version of a freshly minted local Docker image.

For the environment parameter, I pass "CONN_STR", the full connection string to the CockroachDB database. Note that instead of "localhost", I'm passing "cockroach_test", the name of the CockroachDB docker-compose service.

For the command parameter, I pass "./crdb-test -db". This tells docker-compose what to run from within the test container's working directory. The "-db" flag tells our test executable to run database integration tests.

For the depends_on parameter, I pass "cockroach_test", telling docker-compose that we want to wait until CockroachDB is up before trying to execute our tests.

Finally, to run everything, we call the docker-compose CLI passing the name of our docker-compose file ("crdb-test.yaml" in this case) and a few additional flags you might find helpful, including --abort-on-container-exit, which tears down all containers when one of them exists, cleaning everything up once our tests are run and passing our exit code back to the caller.

$ docker-compose --no-ansi -f crdb-test.yaml up --abort-on-container-exit --force-recreate
Recreating cockroachdb-testing_cockroach_test_1 ... 
Recreating cockroachdb-testing_cockroach_test_1 ... done
Recreating cockroachdb-testing_app_test_1       ... 
Recreating cockroachdb-testing_app_test_1       ... done
Attaching to cockroachdb-testing_cockroach_test_1, cockroachdb-testing_app_test_1
cockroach_test_1  | WARNING: no logs are available with the 'none' log driver
app_test_1        | run database tests true
app_test_1        | PASS
cockroachdb-testing_app_test_1 exited with code 0
Aborting on container exit...
Stopping cockroachdb-testing_cockroach_test_1   ... 
Stopping cockroachdb-testing_cockroach_test_1   ... done