Succeeding with the 3 Musketeers pattern

Photo by Recha Oktaviani on Unsplash

Contents

• Why use the 3 Musketeers?

• What is the 3 Musketeers pattern?

• When can you use the 3 Musketeers?

• An example of using the 3 Musketeers pattern

Why use the 3 Musketeers?

The 3 Musketeers pattern has, to me, three (funnily enough!) primary reasons to use it:

Consistency

All environments you use it in should(!) behave the same. Developer machines, CI/CD pipelines, etc. Debugging issues that arise, such as in your pipeline, is considerably more accessible when it behaves identically locally. Rather than pushing code up and waiting for your pipeline to tell you it's incorrect, you can test it out locally first in an identical manner and save yourself some time.

Ease of use

One command to rule them all - Make doThing. No worrying about every developer fully understanding the madness under the hood. Your onboarding Engineers can get straight into the thick of it, and your team can easily embrace your latest addition.

Speed

Development is much quicker when you work consistently across the whole team, ideally org. When you have a pattern to follow for new commands, there's no thinking about exactly how to implement it - you follow the pattern. Following the pattern speeds up others, too; they don't have to learn a new toolchain for every new command.

What is the 3 Musketeers pattern?

I won't do a better job than this website, but I'll take a go at summarising here for those wanting a TL; DR.

'3 Musketeers' is a method that aims to keep your development repeatable and consistent by using three tools: (GNU) Make, Docker, and Docker Compose.

Using Make as an orchestrator and entry-point, you can use docker/compose to provide the environment where you then execute your code.

I’ve also recently come across a replacement to Make (as this isn’t Make’s real purpose) called Just. It seems to focus more on why we use Make in the 3 Musketeers pattern, so maybe it will replace it one day.

When can you use the 3 Musketeers?

So now that we understand what the pattern is and the benefits of it, when should we use it?

All the time. (Mostly). The 3 Musketeers is best when used for commands or functional tasks.

Its primary use case is for making your build pipelines simpler to follow and easy to develop locally. Any time you have a step that runs in your CI/CD pipeline, you should define it as a Make command and follow the pattern. Creating and updating the step behaves the same locally as your pipeline, so it should be a quick iterative process.

Another great scenario is whenever you need to run services in conjunction - such as for local testing. Have a microservice architecture and want to run all the services simultaneously to develop a new feature? Use the musketeers.

Onboarding a new Engineer to the team can be tricky and convoluted. Why not set up the environment using the 3 Musketeers? You can host most services (such as Databases) in a container, spin up the service they will work on, and then seed it with a bash script - all from one command.

Any other tooling scenarios are great candidates, too. Seeding, resetting, uploading, testing, you name it.

When not to use it

Sometimes, it’s really easy to get lost in the weeds with a new pattern and try to use it everywhere. Looking at you, design patterns.

While I highly recommend the 3 Musketeers for many things, don’t use it for everything. That’s an anti-pattern.

It's a lot less useful when it’s just one or two people working on a project or there’s a project with not many repeated ‘commands’.

The 3 Musketeers is also not great if all it’s doing is overcomplicating things. If all you’re doing is adding another place to run commands from, maybe don’t add them. For example - If you’re just in a Javascript environment (where you likely use Yarn/Node/NPM) and all of your commands for building, testing, etc., are already in a ‘package.json’ file, then unless you plan to add more automation across the project outside of your javascript I would just leave it be. You’re just adding confusion and another place that needs updating.

An example of using the 3 Musketeers pattern

We primarily use the pattern for our build pipelines, as it enables running and testing them locally before committing and pushing code up to the repository.

Let's review an example command for running tests on a .NET project in the build pipeline. I’ll omit the pipeline file to keep this as minimal as possible, but I’m afraid a small amount of prerequisite knowledge around Docker/compose will be required, and you’ll need to install Make and Docker if you want to do a hands-on follow along.

Here's a gist link for those who want to see the example pieced together. Here’s an example .NET project which you can follow along with.

So, the premise for the example is this: 'I want to run the unit tests for my .NET project in a CI/CD pipeline and locally as part of my development cycle.'. A very common scenario.

First, we need to figure out how to run our unit tests. It's very straightforward with .NET, luckily. In a realistic scenario, we would add extra things, such as static analysis, but I want to keep this simple enough. In my team, we would usually add this in by extrapolating the command into a Bash script.

So, to run unit tests in dotnet, we simply need to run this command in a terminal (with the .NET SDK installed):

dotnet test

That’s straightforward. It works on my machine; it's all good, right? Commit, push, watch it run…

Oh, wait, our build pipeline doesn’t know the dotnet command. It doesn’t have the .NET SDK installed, and I can’t install dependencies on the agent. Hmm. Now what? Dockerise it!

So, let’s use Docker to provide a consistent environment. Microsoft offers an official Docker image with the .NET SDK, which we’ll use. We’ll put it in a ‘docker-compose.yaml’ file like so:

version: "3.8"

services:
    dotnet:
        image: mcr.microsoft.com/dotnet/sdk:latest
        working_dir: /app/app/src
        volumes:
            - .:/app

So, we’ve added the ability to spin up a container with .NET installed and mounted our source code to the container when it runs.

Now, we could just call it a day there and, in our pipeline, run this command:

docker-compose run --rm dotnet dotnet test

But let’s simplify it for everybody as there’s a bit to unpack there, and a big advantage of Make is simplicity.

So, let’s create our Make file. I’ll call it ‘Makefile’. Here’s the first draft:

test:
    echo "Running Make test"
    docker compose run --rm dotnet dotnet test

Now, all we have to do in our terminal and pipeline is run the following:

make test

Isn’t that so much simpler? You can add that to your team documentation, where it will likely never change. If you had added the previous command, what happens when you want to change the behaviour? You might need to extend your test command to perform static code analysis, but then you’ll have to update your docs and pipeline and ensure your team knows they must run the commands the new way.

Once your Makefile expands to contain other commands, particularly similar ones, you might want to tidy up and follow some best practices. I’ve expanded the Makefile in the linked gist to include a build command and a slight refactor to improve the file. I’ve also extended it to utilise a bash script so you can see how to utilise it in a similar scenario.

In my team, we utilise Bash scripts alongside the 3 Musketeers. We use Bash scripts because most Docker images can execute them (maybe with a tweak), we want to do several things in a command, and we are comfortable with Bash.

I suspect a more traditional use of the pattern is to chain multiple Make commands together instead. The expanded gist does both of these.

In Summary

You might not need the 3 Musketeers; it has its downsides - including added complexity in small projects. But, if you have yet to use it or want some order to your chaos, I recommend trying it.

You don't need to use it for everything - we don't, but I strongly suggest it for your CI/CD pipelines at the minimum and anything automated that is repeatedly needed in your local development process.