DEV Community

Cover image for Test that forking code!
Paul Cochrane šŸ‡ŖšŸ‡ŗ
Paul Cochrane šŸ‡ŖšŸ‡ŗ

Posted on • Originally published at peateasea.de on

Test that forking code!

Originally published on peateasea.de.

Not only should the final commit in a pull request patch series pass all its tests but also each commit back to where the pull request branched from main. A Git alias helps to automate this process. Introducing: git test-fork.

I take pride in delivering quality code in my pull requests, be it either for clients,1 or for open source software submissions. One aspect of that quality is to deliver small chunks of working functionality in each commit. To ensure that a pull request has this property, I like to ensure that each commit builds correctly, that the test suite passes and, depending on the project, that the linter checks also all pass. Of course, for a multi-commit pull request checking each commit manually is a lot of work. To avoid unnecessary work, Iā€™ve automated this process with a Git alias: git test-fork. Hereā€™s why I use it and how its inner workings, erm, work.

My main focus: quality

Have you ever been doing software archaeology on a codebase and wondered why the test suite on an old commit suddenly no longer works? And have you then spent the next several hours trying to work out what the test failures have to do with the bug only to find out that theyā€™re not related? If so, you know how annoying if not downright painful such a situation is. Also, youā€™ll know how stressful this can be because youā€™re usually under pressure when bug-hunting and wild goose chases are the last thing you need. Wouldnā€™t it have been nice if those who came before you2 had ensured that the test suite passed at every commit? This would have saved you time, stress and grey hairs. Also, this would have reduced friction in your bug-hunting, letting you find (and hopefully solve!) the bug earlier. Instead, you went on a minor odyssey, delving into problems unrelated to those you were actually trying to solve.

Iā€™ve had that kind of experience one too many times in the past. My solution: make sure that each and every commit in a pull request passes its test suite and, if relevant, passes the projectā€™s linter tests. Before I show the command and explain how its internals work, why do I think going to this effort is worth it?

To me, a passing test suite is a sign of quality and that whoever wrote the code cared enough to submit robust, well-tested code. Itā€™s sort of like a craftsperson taking the time to build in quality and taking pride in their work and a job well done. Software isnā€™t a work of art in the way that Biedermeier furniture or a Stradivarius violin is, however a strong focus on quality and craftsmanship (for want of a better word) makes for better software.

A codebase in which each commit passes its tests has technical benefits: itā€™s possible to use git bisect automatically and any commit could be pushed to production at any time, affording a high level of development flexibility and responsiveness.

Personally, I like the feeling of a solid foundation; itā€™s something I know I can depend upon and build upon.

Swings and roundabouts

Of course, running the full test suite on each commit has its downsides. It slows you down and some might feel that it adds unnecessary friction to the development workflow, especially when the test suite is slow. Others might feel that itā€™s just another exercise in gold plating or some kind of over-the-top obsessive programmer behaviour.

These are valid points and there needs to be a balance between a desire for high quality and getting code ā€œout the doorā€. After all, one can go too far and being extremist about things is usually a bad sign.

That being said, sometimes itā€™s a good idea to slow down when developing software so that our brains can mull over what weā€™re doing and contemplate the bigger picture. There have been times when Iā€™ve been running the tests on each commit for a given branch and have realised that a commit wasnā€™t necessary, or the idea behind a direction of development was plain wrong. This extra cogitation time allowed me to rethink what I was doing and ultimately led to better software. Also, by not submitting some code, I saved my colleaguesā€™ time, because it was code they didnā€™t have to review!

Sometimes, ensuring that each commit passes the test suite along a feature branch picks up on things Iā€™d missed during development and should have fixed. Recently, I was refactoring some code and had finished a long-ish feature branch. I ran the tests together with the linter checks and the linter spotted a bug: while renaming a module Iā€™d not updated the imports in a file. This was the code ā€œtalkingā€ to me. It showed that Iā€™d missed a particular code change and that there was a hole in my test coverage. This was a big win because it gave me the opportunity to improve the tests which will reduce risk and friction when refactoring in the future.

Also, if you notice that it takes ages to test each commit on a feature branch, then this is not a hint that you shouldnā€™t be testing each commit, but a sign that the test suite is too slow. Thatā€™s something that you could invest time in in the future. Itā€™s like an application of ā€œif it hurts, do it more oftenā€.

Another criticism of this technique is that it takes a long time on branches with many commits. This is a good thing: it provides feedback to you to keep your pull requests and feature branches small and focused. Again, ā€œif it hurts, do it more oftenā€!

Automatically testing forks via Git

Obviously, testing all commits on a fork of the main branch isnā€™t something one wants to do manually. A single commit? Fine. Ten commits on a feature branch? Nah, Iā€™ll pass, thanks. šŸ˜ƒ

So how do you know when the current branch forked from the main branch? And how do you make Git run tests on each commit? Letā€™s get to that now.

Hereā€™s the alias I have set up in my .gitconfig:

test-fork = !"f() { \
  [x$# = x1] && testcmd=\"$1\" || testcmd='make test'; \
  upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1); \
  current_branch=$(git rev-parse --abbrev-ref HEAD); \
  fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch); \
  git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"; \
}; f"
Enter fullscreen mode Exit fullscreen mode

Thereā€™s a lot going on in here, so I tried representing it as a diagram:

whichā€“along with the detailed explanations belowā€“I hope aids understanding of the alias code above.

A big shell function

Letā€™s focus on the full test-fork alias. In essence, this command uses git rebase to execute a command (via the -x option) on each commit from a given base commit.

This is a very long command, and as far as Git is concerned, this command is all one line. However, to make it easier to edit within the .gitconfig file, Iā€™ve split it across several lines by using trailing backslashes (\).

On the left-hand side of the equals sign is the name of the alias: test-fork. On the right-hand side is where all the action is happening: a shell function that Git runs when the user enters the command

$ git test-fork
Enter fullscreen mode Exit fullscreen mode

within a Git repository.

We define the shell function within a large double-quoted string, and hence we have to be careful when embedding double quotes. The exclamation mark at the beginning means that Git treats the alias as a shell command and hence it wonā€™t prefix the alias by the git command as would be the case without the exclamation mark.

The shell function has this form:

f() { ... code ... }; f
Enter fullscreen mode Exit fullscreen mode

meaning that we define the function and then immediately run it. The semicolon separates the function definition from its call; by entering f at the end we call the function that we just defined.

Defining the test command to run

The first line of the function is

[x$# = x1] && testcmd=\"$1\" || testcmd='make test'
Enter fullscreen mode Exit fullscreen mode

This code tests to see if we have an argument and if so, sets the variable testcmd to its value (i.e. the variable $1). Otherwise, we set testcmd to the default value of make test. In other words, if you run

$ git test-fork 'some-test-command'
Enter fullscreen mode Exit fullscreen mode

then some-test-command will test each commit in your branch. If you donā€™t specify a command explicitly, then the alias falls back to using make test.

The variable $# is a special parameter in Bash and expands to the number of positional parameters. In other words, if there is a single argument, $# will be the value 1 and the test

[x$# = x1]
Enter fullscreen mode Exit fullscreen mode

will evaluate as true. The code will then take the first branch of this implicit if block (i.e. the bit after &&) setting testcmd to the value passed in on the command line and using this as the test command for the rest of the code in the shell function.

The origin of branches

The next line in the function is

upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1)
Enter fullscreen mode Exit fullscreen mode

which determines the name of the upstream (a.k.a. origin) branch from which the local feature branch is based. This information will later help us work out where the feature branch forked off from the main line of development. This is the first Git-related command, which Iā€™ve indicated by the number ā‘  and the colour blue in the diagram above.

This line runs the command within the $() and returns its result as the value of the variable. The $() is a Bash feature called command substitution and

allows the output of a command to replace the command itself.

The command

$ git branch --remotes
Enter fullscreen mode Exit fullscreen mode

returns a list of all locally-known remote branches. The one weā€™re interested in is either origin/master or origin/main, or what origin/HEAD is currently pointing to.

The output from git branch --remotes can be different in certain situations. Usually, you will see output like the following:

$ git branch --remotes
  origin/HEAD -> origin/master
  origin/master
  origin/rename-blah-to-fasl
  origin/refactor-foo-baa
Enter fullscreen mode Exit fullscreen mode

where we have a reference origin/HEAD which points to the actual upstream main branch, which in this case is origin/master.

In other situations (and I havenā€™t been able to work out exactly why; I think this has to do with sharing an upstream repository with others, but Iā€™m not sure), the output omits a pointer from origin/HEAD to the main remote branch, giving e.g.

$ git branch --remotes
  origin/main
Enter fullscreen mode Exit fullscreen mode

where Iā€™ve used the now more common main name for the main branch in the upstream repository.

The filtering commands after git branch --remotes handle both situations. By filtering on HEAD, master or main with the grep command, we ensure that all variations are in the filtered output. The cut extracts the reference that origin/HEAD is pointing to (if origin/HEAD exists in the output) and the head ensures we only select the first entry should there be multiple matches. If origin/HEAD doesnā€™t exist in the output, the cut passes its output to head and we again select the first entry in the list of appropriate upstream branch names.

When constructing grep regular expressions in the shell, we have to escape Boolean-or expressions (the pipe character |) and groups (parentheses, ()) with a backslash (\). In the case we have here, we have to ā€œescape the escape characterā€ within the Git alias by using two backslashes (\\). Thus, when Git passes the command to the shell, there is only a single backslash character present and the regular expression is formed correctly.

After all this hard work, the variable upstream_base_branch contains the name of the upstream base branch.

Where are we now?

The third line in our shell function

current_branch=$(git rev-parse --abbrev-ref HEAD)
Enter fullscreen mode Exit fullscreen mode

finds out the name of the current branch. I.e. this is the name of the feature branch that we want to test. Iā€™ve highlighted this information by the number ā‘” and the colour green in the diagram above.

We use this information, along with the upstream branchā€™s name, to work out where the current branch forked from the main line of development. This is the purpose of the next line.

Where did we forkā€™n come from?

Now weā€™re in a position to work out where the feature branch forked3 from the main line of development. In particular, we want to find the commit id of this fork point. Hence the next line of code assigns a variable called fork_point:

fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch)
Enter fullscreen mode Exit fullscreen mode

The git merge-base command

finds the best common ancestor(s) between two commits to use in a three-way merge.

When using the --fork-point option, the command takes the form

git merge-base --fork-point <ref> [<commit>]
Enter fullscreen mode Exit fullscreen mode

The --fork-point option is key for us here because it finds

the point at which a branch (or any history that leads to <commit>) forked from another branch (or any reference) <ref>.

In our case, we use git merge-base to find the commit at which the current_branch diverged from upstream_base_branch. Iā€™ve denoted this with the number ā‘¢ and the colour red in the diagram above.

It was a fair bit of work to get to this point, but now weā€™re in a position to use git rebase to run our tests.

Skip to the commit, my darling

Now we get to the very heart of the matter: iterating over each commit in the branch and running the test command on each commit as we go. Iā€™ve referenced this process by the orange number ā‘£ and arrows in the diagram above.

We run the test command via the -x/--exec argument to git
rebase
:

git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"
Enter fullscreen mode Exit fullscreen mode

Here, we rebase the branch we are currently on (i.e. the feature branch) onto where it forked from the upstream base branch. Note that git rebase operates on a reference which exists upstream and aborts the rebase if there is no upstream configured. This is why we use the upstream base branch when working out the fork point. Also, the upstream branch is usually the branch used for comparisons to the feature branch when submitting a pull request on systems such as GitHub, GitLab, Gitea, etc. Thus it makes sense to consider the state of the upstreamā€™s main branch rather than the local main branchā€™s current state.

The -x option takes a string argument of the shell command to run. In our case here, we need to escape the double quotes so that they donā€™t conflict with the quotes enclosing the entire alias code. Doing so ensures that quotes still enclose the command to run for each commit in the rebase process. Note that we need double quotes here as well so that the value of $testcmd is interpolated into the command ultimately executed by git rebase.

To provide context for the test command, and to indicate where the git rebase process currently is, we precede it with

git log -1 --oneline
Enter fullscreen mode Exit fullscreen mode

This prints the abbreviated commit id and subject line on a single line for the current commit, which can be useful to know if the test command fails.

Finally, we run the test command defined in the variable $testcmd. This is either the command specified as an argument to git test-fork or is make test if no arguments were given.

Thatā€™s it!

Thatā€™s the guts of the test-fork alias in detail.

Clear as mud? Ok, letā€™s see the command in action and hopefully its use and utility will make more sense.

git test-fork in action

Sometimes itā€™s easier to understand whatā€™s going on if one sees something run. I canā€™t do that dynamically here, but I can show a representative example.

A quick but detailed example

Hereā€™s an example from a Python project where I was wanting to reduce technical debt with the aid of the pylint code checker. Iā€™d used pylint to sniff out any code smells which might need addressing and had a few commits on a feature branch which had fixed these issues. I now wanted to make sure that Iā€™d not broken anything in the process, hence I wanted to run the test suite on each commit in the feature branch. Since I have a Makefile which wraps the actual test command behind a simple test target, I only needed to run git test-fork. This is the output:

$ git test-fork
Executing: git log -1 --oneline; make test
8e6ff5b (HEAD) Fix import ordering
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 239.65s (0:03:59) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
2c5c1d9 (HEAD) Remove unnecessary "dunder" calls
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 248.33s (0:04:08) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
bccc026 (HEAD) Remove reimported module
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================

<snip-lots-of-test-output>

======================= 257 passed in 254.63s (0:04:14) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Successfully rebased and updated refs/heads/address-technical-debt.
Enter fullscreen mode Exit fullscreen mode

There are a few things to note about the output:

  • git rebase echoes the command itā€™s running: Executing: git log -1 --oneline; make test. This lets us check that Git is running the command we want it to run.
  • We see the git log -1 --oneline output fairly clearly. Bash displays this with nice, bright colours and is easy to see in the terminal. Unfortunately, I couldnā€™t reproduce that here though. Sorry. šŸ˜•
  • make changes into the src/ directory to then run the test target within that directory: make --directory=src test and make[1]: Entering directory '/home/cochrane/a-python-project/src'.
  • Now we see the pytest invocation that make test runs.
  • There is lots of output from pytest. Iā€™ve removed a lot so we can focus on the main points in this discussion.
  • We see that the tests passed, yay! šŸŽ‰
  • make returns to the original directory after the commands in the test target have completed successfully: make[1]: Leaving directory '/home/cochrane/a-python-project/src'.
  • Git checks out the next commit on the feature branch and the process repeats.

Rebase starts from the base

Note that the rebase command starts running from the projectā€™s base directory. This is independent of where you run the git test-fork command. So, if you want to run pytest on an individual file, youā€™ll need to explicitly change into the appropriate directory as part of the test command. In other words, if you have to be in the src/ directory to run a single test like this:

$ pytest tests/test_views.py
Enter fullscreen mode Exit fullscreen mode

then using

$ git test-fork 'pytest tests/test_views/py'
Enter fullscreen mode Exit fullscreen mode

wonā€™t work, because git rebase operates from the base directory and pytest wonā€™t be able to find the files. Also, using the full path to the test file by running

$ git test-fork 'pytest src/tests/test_views/py'
Enter fullscreen mode Exit fullscreen mode

probably wonā€™t work. At least, it doesnā€™t work in my case because I have a pytest.ini file which pytest reads and itā€™s in the src/ dir. Thus, the only command that will allow you to run individual test files in such a situation is:

$ git test-fork 'cd src && pytest tests/test_file.py'
Enter fullscreen mode Exit fullscreen mode

where each command execution by git rebase also changes into the required directory to run the individual test file. Running this command has output much like the detailed output included above.

Common usage variations

Other common invocations I use are:

$ git test-fork 'make lint'
Enter fullscreen mode Exit fullscreen mode

which runs the linter checks on the entire codebase for each commit in the branch.

Also, I tend to use this one a lot:

$ git test-fork 'make test && make lint'
Enter fullscreen mode Exit fullscreen mode

which runs the full test suite and the linter checks for each commit in the feature branch. Note that we can chain commands in the argument passed to git test-fork by using the Bash Boolean-and operator: &&.

What to do if things go wrong

Nobodyā€™s perfect and something could go wrong in the middle of the rebase process. Actually, this is exactly what weā€™re trying to do: we want to sniff out any problems before they make their way upstream into a pull request. This way we avoid our colleagues from having to stumble across problems when reviewing the code.

So what happens if the test suite fails in the middle of a rebase? Git interrupts the rebase. Thatā€™s all. Actually, itā€™s great: youā€™re dropped right into the middle of where the problem is, which is the best place to be able to fix it.

After fixing the issue, run make test (or the equivalent command) to check that everything now works. Add the changes with

$ git add -p
Enter fullscreen mode Exit fullscreen mode

or use git add on each of the files. Then itā€™s simply a matter of amending the commit

$ git commit --amend
Enter fullscreen mode Exit fullscreen mode

before then continuing the rebase with:

$ git rebase --continue
Enter fullscreen mode Exit fullscreen mode

Or, if things look to be too complicated and you might need some thinking time, just abort:

$ git rebase --abort
Enter fullscreen mode Exit fullscreen mode

Then, take a step back, take a deep breath, and dig in again.

Wrapping up

The test-fork alias can be really helpful in finding test or linting issues locally before pushing code to colleagues or collaborators.

Iā€™m fairly sure this code could be improved upon. Still, it works well for my purposes and is a standard part of my process to provide high-quality work to internal teams and external customers.

So what are you waiting for? Go test that forking code!

  1. Iā€™m available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and letā€™s discuss how I can help solve your businessā€™ hairiest problems. ā†©

  2. Of course, ā€œtheyā€ could have been an earlier you! ā†©

  3. I could have said ā€œbranchedā€ here, but the phrase ā€œa branch branchedā€ sounded a bit odd. ā†©

Top comments (0)