Eat Your Vegetables

About a year ago, I was starting a new project at work using Fortran. For anybody who didn't know, the tools and libraries available to Fortran programmers are few and far between, and not particularly extensible or adaptable. But, being the testing and automation proponent I am, I went looking for a testing framework.

The only decent, open-source option I found was called fruit. It wasn't particularly well documented, but it purported to be able to find your tests suites at compile time, so you wouldn't have to manually maintain a driver program. It used Rake as a build system, and the Fortran part was contained in a single source file.

It was unmaintained, and only available from SourceForge, so I figured if I was going to use it, I better take over its maintenance. I grabbed a copy, put it in git, and got to work. I've since cleaned up all the build system components, and even fixed a few of the bugs. But I've made very little changes to the testing framework itself. The framework itself can be found here, and an example hello world type example of how to use it can be found here.

Having done all this, I was terribly unsatisfied with it. It's very imperative by nature, makes heavy use of internal state, and it's output leaves a lot to be desired. Having been maintainer of fruit for about a year now, having seen plenty of other testing frameworks in use, and having been bitten by the functional programming bug, I thought, surely I could write a better testing framework. How hard could it be.

And so, I announce to you now, a new Fortran testing framework: Vegetables. For a healthier code base, be sure to eat your vegetables. I've still got lots to do, but I believe the foundation is sound and proven at this point. I wanted to explain a bit about how I put it together.

My first couple attempts (which you can find in the git history if you are truly curious) just didn't work out. It turns out, Fortran just can NOT handle a recursive data structure properly. So I ended up having to link to C++. This isn't particularly ideal, but it was the only way I could get it to work.

Basically all of the heavy lifting of the framework happens on the C++ side. But all of the interactions happen on the Fortran side by exposing equivalent types and methods. The Fortran types just hold onto a C++ pointer, and pass it back to functions which call the appropriate methods in C++. This means, that unless you really start looking into how the framework is constructed, you won't really be able to tell. A user won't have to concern themselves with dealing with the C++ side of things.

There is downside to doing things this way. I've basically made a memory leak. The good news, it's memory that under normal circumstances wouldn't be freed until the end of execution anyway. Basically, the whole process of defining your tests builds up a single data structure containing all of them. Then, running the tests creates a second, identically shaped data structure containing all the results. But neither of these go out of scope until the end of the program, so their destructors wouldn't be called until then anyway. The only time you'd even be able to tell is if you're doing something really weird, or testing the testing framework.

Which brings me to my next point. The framework is fully tested, using itself as the testing framework.

Ok, maybe it's not that mind blowing. I'm sure some other testing frameworks have done this. But it was a little tricky trying to do TDD this way. Oh, yeah, I used TDD to build my testing framework. Yeah, that one does deserve a mind explosion.

At the early stages of the process the tests weren't really run, they only compiled. But they at least existed and trying to write them was the impetus to say which function or type needed to be written next. I had a pretty good idea for the overall design before I started, which really helped too.

The basic strategy/design is simply thus: a test is either a test case, or a collection of tests. Hence, a recursive data structure. Then running a test case produces a test result, and running a test collection produces a collection of results; an identically shaped recursive data structure.

A test collection has a description, and a collection of tests. A test case has a description and a function which returns an assertion result. Assertion results can be .and.ed together, so a result from a test case can even tell how many assertions were made as a part of it.

Enough of the abstract, how do you use this thing. Pretty simple, actually. You just define your tests like so.

function test_passing_case_behaviors() result(test)
    use Vegetables_m, only: TestCollection_t, given, then, when

    type(TestCollection_t) :: test

    test = given("a passing test case", &
            [when("it is run", &
                    [then("it knows it passed", checkCasePasses), &
                    then("it has 1 test case", checkNumCases), &
                    then("it has 1 passing case", checkNumPassingCases), &
                    then("it has no failing case", checkNumFailingCases), &
                    then("it's verbose description still includes the given description",  checkVerboseDescription), &
                    then("it's failure description is empty", checkFailureDescriptionEmpty), &
                    then("it knows how many asserts there were", checkNumAsserts), &
                    then("it has no failing asserts", checkNumFailingAsserts), &
                    then("it knows how many asserts passed", checkNumPassingAsserts)])])
end function test_passing_case_behaviors

given is a function which takes a description and an array of test collections and returns a test collection. when is a function which takes a description and an array of test cases and returns a test collection. then is a function which takes a description and a function, and returns a test case. The function must take no arguments, and return a result from one or more assert functions. Like so:

function checkCasePasses() result(result_)
    use example_cases_m, only: examplePassingTestCase
    use Vegetables_m, only: &
            Result_t, &
            TestCase_t, &
            TestCaseResult_t, &
            operator(.and.), &
            assertNot, &
            assertThat

    type(Result_t) :: result_

    type(TestCase_t) :: test_case
    type(TestCaseResult_t) :: test_result

    test_case = examplePassingTestCase()
    test_result = test_case%run()
    result_ = assertThat(test_result%passed()).and.assertNot(test_result%failed())
end function checkCasePasses

There are also describe and it functions which take a description and an array of test cases, and description and function, respectively if you don't need full BDD style given-when-then. At the top level, a function testThat accepts an array of test collections and returns a single test collection. A single subroutine runTests accepts a test collection as argument, runs it, and reports the results.

The test specification is reported to the user prior to running the tests, using indentation to signify the nesting of the test collections. Then, once the tests are run, any failures can be reported exactly the same way.

The result is a composable, pure-functional, testing framework, with a very nice output as shown below, somewhat abbreviated. See if you can figure out what's broken from the results.

Running Tests

Test that
    A test collection
        can tell how many tests it has
        includes the given description
        includes the individual test descriptions

...

A total of 100 test cases

Failed

2 of 50 cases failed
2 of 59 assertions failed

Test that
    A test collection
        can tell how many tests it has
            Expected '2' but got '4'
    A test case
        Only has 1 test case
            Expected '1' but got '2'

There is still a fair amount of work left to do, but I am now confident that it will work and is something worth using. Check it out here if you're curious, and feel free to ask any questions you might have or make any suggestions.

I also have a question for anybody who might consider using this. Would it be worth supporting a way to report all of the assertions that passed? Right now I've only got the mechanics to show all of the test cases, and the results of any failed assertions, but not the passing assertions. It wouldn't be hard to add, I just haven't decided if it would be worth it. What do you guys think?

EDIT:

I've actually managed to get the framework working using only Fortran. The data structures and functionality are the same as described above, but there's no more C++ that needs to be linked in, making this a single file drop in for anyone that doesn't want to use the same build system and automation as me.