Introduction to Doctests in Haskell
Documenting software can be challenging, but it doesn’t always need to be so.
In this article, we’ll introduce doctests: a concept that makes the documentation process pleasant and effective.
By putting tests inside module docs, doctests allow functions to speak for themselves and help you get the most out of your testing efforts.
Read the article to learn:
- what doctests are;
- how to define them in Haskell;
- which library to use for Haskell doctests.
What are doctests?
Doctests are simply pieces of text embedded in the documentation that look like interactive sessions. With a special library, you can run these sessions and verify that they return the correct value.
The idea for this comes from Python’s awesomedoctest
module, but since then has spread to virtually every programming language.
In Haskell, doctests are GHCi sessions within Haddock comments.
If you ever had to skim through the Haskell source code documentation, you might have noticed those lines with fancy >>>
symbols at the beginning.
-- | @const x@ is a unary function which evaluates to @x@ for all inputs.
--
-- >>> const 42 "hello"
-- 42
--
-- >>> map (const 42) [0..3]
-- [42,42,42,42]
As you might have guessed, those are doctests.
How to define a doctest in Haskell?
Let’s look at an example of a basic doctest.
-- | 1 + 2 is 3.
-- >>> 1 + 2
-- 3
As you can see, Haskell doctests have three requirements:
Every doctest example should be placed within a valid piece of Haddock documentation, which is marked by either
-- |
or{- |
.Every doctest example should start with
>>>
and contain a valid Haskell expression that is in scope (sometimes you have to be explicit about theimport
statements – we’ll cover that further in the article).Every doctest example should be followed by a line containing the expected result of evaluating the expression.
You can find additional info on doctest markup in the readme of the doctest
library.
Now, let’s create a project with doctests and go over Haskell libraries that you can use to run those doctests.
Creating a Haskell project with doctests
To work with doctest libraries, we need to create a Haskell project with doctests.
First, bootstrap a project using stack
:
stack new doctests-demo
After that, go to the project’s root directory and create a Haskell module:
cd doctests-demo
touch src/Sample.hs
Finally, add some functions with doctests to the newly created Sample.hs
module:
-- src/Sample.hs
module Sample where
-- |
-- >>> foo + 13
-- 55
foo :: Integer
foo = 42
-- |
-- >>> bar
-- "bar"
bar :: String
bar = "bar"
The project is now ready for running doctests.
Doctest libraries in Haskell
We’ll cover two doctest libraries in the Haskell ecosystem: doctest
and cabal-docspec
. The first one is older and more popular, the second one is less popular but solves some of doctest
’s issues.
doctest
doctest
is one of the most commonly used and actively maintained doctest libraries. At the same time, it has drawbacks like bad performance for large-scale projects and dependency on GHC as a library.
How to use doctest
First, install the library via stack
:
stack install doctest
After that, go to the project’s root directory and use the library’s executable:
cd doctests-demo
doctest src
The command above should output something like this:
Examples: 2 Tried: 2 Errors: 0 Failures: 0
Cons of using doctest
While it’s an awesome piece of software, the library has some drawbacks.
The biggest drawback is that the library seems to be too slow for large-scale projects. One of the reasons causing performance issues is that the library reloads the source between each group of doctest examples. This is done to avoid example groups influencing each other. You can learn more about this in the library’s readme.
There are also several more minor drawbacks.
First, it has a dependency on GHC as a library, which means it’s very likely that something will eventually break down when you switch the project to a new compiler version.
Second, when used via stack
, you might need to create an additional test suite to bring the project dependencies into scope.
Illustration of the dependency drawback.
Let’s illustrate this by editing our doctest examples a little (don’t forget to add aeson
and text
to the list of dependencies):
-- src/Sample.hs
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TemplateHaskell #-}
module Sample where
import Data.Aeson
import Data.Aeson.TH
import Data.Text
data Anime =
Anime { title :: Text
, rating :: Double
}
$(deriveJSON defaultOptions ''Anime)
-- |
-- >>> encode favourite
-- "{\"title\":\"One-Punch Man\",\"rating\":8.9}"
--
favourite :: Anime
favourite = Anime "One-Punch Man" 8.9
Now, if we try to run the doctests using doctest src
, we’ll fail with the following message:
src/Sample.hs:9:1: error:
Could not find module ‘Data.Aeson’
Perhaps you meant Data.Version (from base-4.14.3.0)
Use -v (or `:set -v` in ghci) to see a list of the files searched for.
|
9 | import Data.Aeson
| ^^^^^^^^^^^^^^^^^
The workaround is to create an additional test suite.
tests:
doctests:
source-dirs: doctests
main: Main.hs
ghc-options:
- -threaded
- -rtsopts
- -with-rtsopts=-N
dependencies:
- doctest
# bring in your project into the scope
# as well as its dependencies
- doctests-demo
After that, you need to add the test suite directory and Main.hs
executable:
mkdir doctests
touch doctests/Main.hs
-- doctests/Main.hs
import Test.DocTest
-- This test suite exists only to add dependencies
main :: IO ()
main = doctest ["src"]
Finally, run the doctests using the following command:
stack test :doctests
cabal-docspec
cabal-docspec
has fewer contributors and less overall community attention, but a bunch of people already use it in real projects.
How to use cabal-docspec
Let’s use cabal-docspec
to run the doctests in our doctests-demo
project.
First, setup cabal-install
and the global compiler. We suggest using ghcup to achieve this.
After that, build the project with Cabal:
cabal v2-build
Then download cabal-docspec
binaries from the release page:
curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20211114/cabal-docspec-0.0.0.20211114.xz > cabal-docspec.xz
xz -d < cabal-docspec.xz > "$HOME"/.local/bin/cabal-docspec
rm -f cabal-docspec.xz
chmod a+x "$HOME"/.local/bin/cabal-docspec
Now, run the doctest examples:
cabal-docspec
The command above should fail with the following error:
expected: "{\"title\":\"One-Punch Man\",\"rating\":8.9}"
but got:
^
<interactive>:10:1: error:
Variable not in scope: encode :: Anime -> t
The above happens because the library handles things a little differently – it requires modules to be explicitly imported/exported.
Let’s edit our doctests a little to make cabal-docspec
work:
-- |
+ -- >>> import Data.Aeson
-- >>> encode favourite
-- "{\"title\":\"One-Punch Man\",\"rating\":8.9}"
--
favourite :: Anime
favourite = Anime "One-Punch Man" 8.9
Now the cabal-docspec
command should succeed:
Total: 2; Tried: 2; Skipped: 0; Success: 2; Errors: 0; Failures 0
Examples: 2; Tried: 2; Skipped: 0; Success: 2; Errors: 0; Failures 0
Why you should use cabal-docspec
Here are the reasons why you might want to use cabal-docspec
for your project:
The library is a lot faster than the alternatives because it uses compiled code.
The library doesn’t depend on GHC as a library, so it’s much more resilient to GHC version changes.
The library doesn’t require recompiling the source code if you only change the doctests – this saves additional time for developers and makes writing documentation pleasant.
All of the above makes it more than a viable alternative.
However, note that the library seems to be closely bound to cabal-install
since it uses cabal-install
generated metadata – the plan.json
file.
Conclusions
Thanks for reading!
In the article, we looked at the concept of doctests, learned how to define them in Haskell, and gave a brief overview of two libraries that can help us verify them: doctest
and cabal-docspec
.
In the next part of this series, we’ll cover the configuration and internals of cabal-docspec
in more detail. To stay updated, follow us on Twitter or subscribe to the newsletter via the form below.
Top comments (0)