Test that forking code!

Originally published on peateasea.de.

Not only should the final commit in a pull request patch series pass all its tests but also each commit back to where the pull request branched from main. A Git alias helps to automate this process. Introducing: git test-fork.

I take pride in delivering quality code in my pull requests, be it either for clients,¹ or for open source software submissions. One aspect of that quality is to deliver small chunks of working functionality in each commit. To ensure that a pull request has this property, I like to ensure that each commit builds correctly, that the test suite passes and, depending on the project, that the linter checks also all pass. Of course, for a multi-commit pull request checking each commit manually is a lot of work. To avoid unnecessary work, I’ve automated this process with a Git alias: git test-fork. Here’s why I use it and how its inner workings, erm, work.

My main focus: quality

Have you ever been doing software archaeology on a codebase and wondered why the test suite on an old commit suddenly no longer works? And have you then spent the next several hours trying to work out what the test failures have to do with the bug only to find out that they’re not related? If so, you know how annoying if not downright painful such a situation is. Also, you’ll know how stressful this can be because you’re usually under pressure when bug-hunting and wild goose chases are the last thing you need. Wouldn’t it have been nice if those who came before you² had ensured that the test suite passed at every commit? This would have saved you time, stress and grey hairs. Also, this would have reduced friction in your bug-hunting, letting you find (and hopefully solve!) the bug earlier. Instead, you went on a minor odyssey, delving into problems unrelated to those you were actually trying to solve.

I’ve had that kind of experience one too many times in the past. My solution: make sure that each and every commit in a pull request passes its test suite and, if relevant, passes the project’s linter tests. Before I show the command and explain how its internals work, why do I think going to this effort is worth it?

To me, a passing test suite is a sign of quality and that whoever wrote the code cared enough to submit robust, well-tested code. It’s sort of like a craftsperson taking the time to build in quality and taking pride in their work and a job well done. Software isn’t a work of art in the way that Biedermeier furniture or a Stradivarius violin is, however a strong focus on quality and craftsmanship (for want of a better word) makes for better software.

A codebase in which each commit passes its tests has technical benefits: it’s possible to use git bisect automatically and any commit could be pushed to production at any time, affording a high level of development flexibility and responsiveness.

Personally, I like the feeling of a solid foundation; it’s something I know I can depend upon and build upon.

Swings and roundabouts

Of course, running the full test suite on each commit has its downsides. It slows you down and some might feel that it adds unnecessary friction to the development workflow, especially when the test suite is slow. Others might feel that it’s just another exercise in gold plating or some kind of over-the-top obsessive programmer behaviour.

These are valid points and there needs to be a balance between a desire for high quality and getting code “out the door”. After all, one can go too far and being extremist about things is usually a bad sign.

That being said, sometimes it’s a good idea to slow down when developing software so that our brains can mull over what we’re doing and contemplate the bigger picture. There have been times when I’ve been running the tests on each commit for a given branch and have realised that a commit wasn’t necessary, or the idea behind a direction of development was plain wrong. This extra cogitation time allowed me to rethink what I was doing and ultimately led to better software. Also, by not submitting some code, I saved my colleagues’ time, because it was code they didn’t have to review!

Sometimes, ensuring that each commit passes the test suite along a feature branch picks up on things I’d missed during development and should have fixed. Recently, I was refactoring some code and had finished a long-ish feature branch. I ran the tests together with the linter checks and the linter spotted a bug: while renaming a module I’d not updated the imports in a file. This was the code “talking” to me. It showed that I’d missed a particular code change and that there was a hole in my test coverage. This was a big win because it gave me the opportunity to improve the tests which will reduce risk and friction when refactoring in the future.

Also, if you notice that it takes ages to test each commit on a feature branch, then this is not a hint that you shouldn’t be testing each commit, but a sign that the test suite is too slow. That’s something that you could invest time in in the future. It’s like an application of “if it hurts, do it more often”.

Another criticism of this technique is that it takes a long time on branches with many commits. This is a good thing: it provides feedback to you to keep your pull requests and feature branches small and focused. Again, “if it hurts, do it more often”!

Automatically testing forks via Git

Obviously, testing all commits on a fork of the main branch isn’t something one wants to do manually. A single commit? Fine. Ten commits on a feature branch? Nah, I’ll pass, thanks. 😃

So how do you know when the current branch forked from the main branch? And how do you make Git run tests on each commit? Let’s get to that now.

Here’s the alias I have set up in my .gitconfig:

test-fork = !"f() { \
  [x$# = x1] && testcmd=\"$1\" || testcmd='make test'; \
  upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1); \
  current_branch=$(git rev-parse --abbrev-ref HEAD); \
  fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch); \
  git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"; \
}; f"

There’s a lot going on in here, so I tried representing it as a diagram:

which–along with the detailed explanations below–I hope aids understanding of the alias code above.

A big shell function

Let’s focus on the full test-fork alias. In essence, this command uses git rebase to execute a command (via the -x option) on each commit from a given base commit.

This is a very long command, and as far as Git is concerned, this command is all one line. However, to make it easier to edit within the .gitconfig file, I’ve split it across several lines by using trailing backslashes (\).

On the left-hand side of the equals sign is the name of the alias: test-fork. On the right-hand side is where all the action is happening: a shell function that Git runs when the user enters the command

$ git test-fork

within a Git repository.

We define the shell function within a large double-quoted string, and hence we have to be careful when embedding double quotes. The exclamation mark at the beginning means that Git treats the alias as a shell command and hence it won’t prefix the alias by the git command as would be the case without the exclamation mark.

The shell function has this form:

f() { ... code ... }; f

meaning that we define the function and then immediately run it. The semicolon separates the function definition from its call; by entering f at the end we call the function that we just defined.

Defining the test command to run

The first line of the function is

[x$# = x1] && testcmd=\"$1\" || testcmd='make test'

This code tests to see if we have an argument and if so, sets the variable testcmd to its value (i.e. the variable $1). Otherwise, we set testcmd to the default value of make test. In other words, if you run

$ git test-fork 'some-test-command'

then some-test-command will test each commit in your branch. If you don’t specify a command explicitly, then the alias falls back to using make test.

The variable $# is a special parameter in Bash and expands to the number of positional parameters. In other words, if there is a single argument, $# will be the value 1 and the test

[x$# = x1]

will evaluate as true. The code will then take the first branch of this implicit if block (i.e. the bit after &&) setting testcmd to the value passed in on the command line and using this as the test command for the rest of the code in the shell function.

The origin of branches

The next line in the function is

upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1)

which determines the name of the upstream (a.k.a. origin) branch from which the local feature branch is based. This information will later help us work out where the feature branch forked off from the main line of development. This is the first Git-related command, which I’ve indicated by the number ① and the colour blue in the diagram above.

This line runs the command within the $() and returns its result as the value of the variable. The $() is a Bash feature called command substitution and

allows the output of a command to replace the command itself.

The command

$ git branch --remotes

returns a list of all locally-known remote branches. The one we’re interested in is either origin/master or origin/main, or what origin/HEAD is currently pointing to.

The output from git branch --remotes can be different in certain situations. Usually, you will see output like the following:

$ git branch --remotes
  origin/HEAD -> origin/master
  origin/master
  origin/rename-blah-to-fasl
  origin/refactor-foo-baa

where we have a reference origin/HEAD which points to the actual upstream main branch, which in this case is origin/master.

In other situations (and I haven’t been able to work out exactly why; I think this has to do with sharing an upstream repository with others, but I’m not sure), the output omits a pointer from origin/HEAD to the main remote branch, giving e.g.

$ git branch --remotes
  origin/main

where I’ve used the now more common main name for the main branch in the upstream repository.

The filtering commands after git branch --remotes handle both situations. By filtering on HEAD, master or main with the grep command, we ensure that all variations are in the filtered output. The cut extracts the reference that origin/HEAD is pointing to (if origin/HEAD exists in the output) and the head ensures we only select the first entry should there be multiple matches. If origin/HEAD doesn’t exist in the output, the cut passes its output to head and we again select the first entry in the list of appropriate upstream branch names.

When constructing grep regular expressions in the shell, we have to escape Boolean-or expressions (the pipe character |) and groups (parentheses, ()) with a backslash (\). In the case we have here, we have to “escape the escape character” within the Git alias by using two backslashes (\\). Thus, when Git passes the command to the shell, there is only a single backslash character present and the regular expression is formed correctly.

After all this hard work, the variable upstream_base_branch contains the name of the upstream base branch.

Where are we now?

The third line in our shell function

current_branch=$(git rev-parse --abbrev-ref HEAD)

finds out the name of the current branch. I.e. this is the name of the feature branch that we want to test. I’ve highlighted this information by the number ② and the colour green in the diagram above.

We use this information, along with the upstream branch’s name, to work out where the current branch forked from the main line of development. This is the purpose of the next line.

Where did we fork’n come from?

Now we’re in a position to work out where the feature branch forked³ from the main line of development. In particular, we want to find the commit id of this fork point. Hence the next line of code assigns a variable called fork_point:

fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch)

The git merge-base command

finds the best common ancestor(s) between two commits to use in a three-way merge.

When using the --fork-point option, the command takes the form

git merge-base --fork-point <ref> [<commit>]

The --fork-point option is key for us here because it finds

the point at which a branch (or any history that leads to <commit>) forked from another branch (or any reference) <ref>.

In our case, we use git merge-base to find the commit at which the current_branch diverged from upstream_base_branch. I’ve denoted this with the number ③ and the colour red in the diagram above.

It was a fair bit of work to get to this point, but now we’re in a position to use git rebase to run our tests.

Skip to the commit, my darling

Now we get to the very heart of the matter: iterating over each commit in the branch and running the test command on each commit as we go. I’ve referenced this process by the orange number ④ and arrows in the diagram above.

We run the test command via the -x/--exec argument to git
rebase:

git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"

Here, we rebase the branch we are currently on (i.e. the feature branch) onto where it forked from the upstream base branch. Note that git rebase operates on a reference which exists upstream and aborts the rebase if there is no upstream configured. This is why we use the upstream base branch when working out the fork point. Also, the upstream branch is usually the branch used for comparisons to the feature branch when submitting a pull request on systems such as GitHub, GitLab, Gitea, etc. Thus it makes sense to consider the state of the upstream’s main branch rather than the local main branch’s current state.

The -x option takes a string argument of the shell command to run. In our case here, we need to escape the double quotes so that they don’t conflict with the quotes enclosing the entire alias code. Doing so ensures that quotes still enclose the command to run for each commit in the rebase process. Note that we need double quotes here as well so that the value of $testcmd is interpolated into the command ultimately executed by git rebase.

To provide context for the test command, and to indicate where the git rebase process currently is, we precede it with

git log -1 --oneline

This prints the abbreviated commit id and subject line on a single line for the current commit, which can be useful to know if the test command fails.

Finally, we run the test command defined in the variable $testcmd. This is either the command specified as an argument to git test-fork or is make test if no arguments were given.

That’s it!

That’s the guts of the test-fork alias in detail.

Clear as mud? Ok, let’s see the command in action and hopefully its use and utility will make more sense.

git test-fork in action

Sometimes it’s easier to understand what’s going on if one sees something run. I can’t do that dynamically here, but I can show a representative example.

A quick but detailed example

Here’s an example from a Python project where I was wanting to reduce technical debt with the aid of the pylint code checker. I’d used pylint to sniff out any code smells which might need addressing and had a few commits on a feature branch which had fixed these issues. I now wanted to make sure that I’d not broken anything in the process, hence I wanted to run the test suite on each commit in the feature branch. Since I have a Makefile which wraps the actual test command behind a simple test target, I only needed to run git test-fork. This is the output:

$ git test-fork
Executing: git log -1 --oneline; make test
8e6ff5b (HEAD) Fix import ordering
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 239.65s (0:03:59) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
2c5c1d9 (HEAD) Remove unnecessary "dunder" calls
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 248.33s (0:04:08) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
bccc026 (HEAD) Remove reimported module
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 254.63s (0:04:14) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Successfully rebased and updated refs/heads/address-technical-debt.

There are a few things to note about the output:

• git rebase echoes the command it’s running: Executing: git log -1 --oneline; make test. This lets us check that Git is running the command we want it to run.
• We see the git log -1 --oneline output fairly clearly. Bash displays this with nice, bright colours and is easy to see in the terminal. Unfortunately, I couldn’t reproduce that here though. Sorry. 😕
• make changes into the src/ directory to then run the test target within that directory: make --directory=src test and make[1]: Entering directory '/home/cochrane/a-python-project/src'.
• Now we see the pytest invocation that make test runs.
• There is lots of output from pytest. I’ve removed a lot so we can focus on the main points in this discussion.
• We see that the tests passed, yay! 🎉
• make returns to the original directory after the commands in the test target have completed successfully: make[1]: Leaving directory '/home/cochrane/a-python-project/src'.
• Git checks out the next commit on the feature branch and the process repeats.

Rebase starts from the base

Note that the rebase command starts running from the project’s base directory. This is independent of where you run the git test-fork command. So, if you want to run pytest on an individual file, you’ll need to explicitly change into the appropriate directory as part of the test command. In other words, if you have to be in the src/ directory to run a single test like this:

$ pytest tests/test_views.py

then using

$ git test-fork 'pytest tests/test_views/py'

won’t work, because git rebase operates from the base directory and pytest won’t be able to find the files. Also, using the full path to the test file by running

$ git test-fork 'pytest src/tests/test_views/py'

probably won’t work. At least, it doesn’t work in my case because I have a pytest.ini file which pytest reads and it’s in the src/ dir. Thus, the only command that will allow you to run individual test files in such a situation is:

$ git test-fork 'cd src && pytest tests/test_file.py'

where each command execution by git rebase also changes into the required directory to run the individual test file. Running this command has output much like the detailed output included above.

Common usage variations

Other common invocations I use are:

$ git test-fork 'make lint'

which runs the linter checks on the entire codebase for each commit in the branch.

Also, I tend to use this one a lot:

$ git test-fork 'make test && make lint'

which runs the full test suite and the linter checks for each commit in the feature branch. Note that we can chain commands in the argument passed to git test-fork by using the Bash Boolean-and operator: &&.

What to do if things go wrong

Nobody’s perfect and something could go wrong in the middle of the rebase process. Actually, this is exactly what we’re trying to do: we want to sniff out any problems before they make their way upstream into a pull request. This way we avoid our colleagues from having to stumble across problems when reviewing the code.

So what happens if the test suite fails in the middle of a rebase? Git interrupts the rebase. That’s all. Actually, it’s great: you’re dropped right into the middle of where the problem is, which is the best place to be able to fix it.

After fixing the issue, run make test (or the equivalent command) to check that everything now works. Add the changes with

$ git add -p

or use git add on each of the files. Then it’s simply a matter of amending the commit

$ git commit --amend

before then continuing the rebase with:

$ git rebase --continue

Or, if things look to be too complicated and you might need some thinking time, just abort:

$ git rebase --abort

Then, take a step back, take a deep breath, and dig in again.

Wrapping up

The test-fork alias can be really helpful in finding test or linting issues locally before pushing code to colleagues or collaborators.

I’m fairly sure this code could be improved upon. Still, it works well for my purposes and is a standard part of my process to provide high-quality work to internal teams and external customers.

So what are you waiting for? Go test that forking code!

1. I’m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and let’s discuss how I can help solve your business’ hairiest problems. ↩

2. Of course, “they” could have been an earlier you! ↩

3. I could have said “branched” here, but the phrase “a branch branched” sounded a bit odd. ↩