quick.Check in Go

There are many types of tests that can be used to verify your software, one of these types of tests are Property-Based Testing (PBT).

PBT's as a concept is over 20 years old and has spawn out of the QuickCheck library in Haskell. This library has been re-implemented in many other programming languages, and in Go we can find it in the testing/quick package.

These tests are usually written side by side with your regular unit tests (often referred as example-based unit tests in PBT literature), and has the power to flush out logical error with the help of large corpus of data.

The core of Property-Based Testing is to use generated (many times with random data) test cases that is validating a property of the system under test.

Lets exemplify this with some Go code and try to do some property-based testing of the calculator example from the previous blog post White box and Black box testing in Go.

package calculator

// Calculator defines a simple calculator with basic operations.
type Calculator struct{}

func (c *Calculator) Add(a, b int) int {
    return a + b
}

func (c *Calculator) Subtract(a, b int) int {
    return a - b
}

func (c *Calculator) Multiply(a, b int) int {
    return a * b
}

func isZero(n int) bool {
    return n == 0
}

func (c *Calculator) Divide(a, b int) int {
    if isZero(b) {
        panic("division by zero")
    }

    return a / b
}

From that post we had this example-based unit test

func TestAdd(t *testing.T) {
    c := Calculator{}
    result := c.Add(1, 2)
    if result != 3 {
        t.Errorf("Expected 3, but got %d", result)
    }
}

which test the Add function with the example of adding 1 with 2.

If we want to test the Add function the property-based way we will need to find a way to test some property of the Add function. In PBT we are going to generate many random tests and therefore we will not have a way to know the correct result of each test. Here is a major pitfall when doing PBT's, the temptation to replicate the implementation in the tests.

We need to look at the Add function with fresh eyes. The Wikipedia page for addition can give us some inspiration if we look closer at the Properties section. We could use one of those properties to create a property-based test for the Add function.

The identity property seems like a good candidate for a PBT.

a + 0 = 0 + a = a

To do this we will first need to create a function that verifies this property. This function should return a bool to indicate if the verification is correct or not. This function will be executed several times with different inputs by the quick.Check function. Here is how the function could look like.

f := func(a int) bool {
        result1 := c.Add(a, 0)
        result2 := c.Add(0, a)

        // a + 0 = 0 + a = a
        return result1 == result2 && result1 == a
    }

The function takes a random integer a to verify the identity element property of the Add function, if correct it will return true.

Now we can take that function and use it in a regular test function in Go.

func TestAdd(t *testing.T) {
    c := Calculator{}

    f := func(a int) bool {
        result1 := c.Add(a, 0)
        result2 := c.Add(0, a)

        // a + 0 = 0 + a = a
        return result1 == result2 && result1 == a
    }

    if err := quick.Check(f, nil); err != nil {
        t.Error(err)
    }
}

The quick.Check function will execute our f function with random input 100 times and check the result in each iteration. The nil value can be replaced with a Config parameter if other than the default values is needed.

To run the the property-based test we only need to use the same command as a regular unit test.

go test

There we have our first PBT.

The quick library is simple and easy to get started with, however there are usually more functionality included in a quickcheck library.

There are usually a shrinking feature available. It has the task to narrow down the input space used for triggering an error (make the function f to return false) when/if that occurs. This can make it much easier to understand what is causing an error.

One other feature are generators, to be able to generate more advanced input to our f function. There are times when we will need something else than a random integer.

If a more advanced library is needed then checkout the gopter library.

PBT's are powerful, but it can take some time to get used to identify good properties to use when writing them. That is also a good thing, you will need to reflect more on the problem you need to test.

The standard Go library is also using the testing/quick library, so that might be a good source for inspiration on how to use it.

Happy testing!