Perfect Elixir: Development Workflows

Today we'll explore the tools and workflows essential for our daily development. Our goal is to create a streamlined onboarding experience and establish efficient mechanisms for code changes. Let's dive into some solutions to see how it all works out.

Table of Contents

• A Reflection on Workflows
  • Goals for Our Workflows
  • Rethinking Branches
• Bootstrapping
  • Be Careful with System Dependencies
  • Maximize Trust
  • The First Step
  • All The Steps
• Daily Workflows
  • In Defence of Shell Scripting
  • Doctor
  • Update
  • Shipit
  • Check All the Things
• Test Automating Our Workflows
  • Testability
  • Mocking System Calls
  • Expect
  • bats! 🦇
• Conclusion

A Reflection on Workflows

It's not uncommon that a team's workflow is "go clone the repo" and "get your pull-requests reviewed". While not inherently bad, this kind of vaguely defined workflow can be difficult to improve for several reasons:

1. Team-wide Pain Points: If running various upgrade commands causes groans, how can we turn these observations into solutions when there's no code to iteratively improve? I've seen teams argue against committing frequently because running the required upgrade-commands was too cumbersome. I've seen developers not want to pull out of fear it might mess up their setup. These anti-patterns are hard to uproot once established.

2. Onboarding Complexity: Without local workflows, onboarding often ends up just a list of manual steps. If new hires encounter a pain point they can probably update the guide, but those changes are likely to just rot themselves. Without code, it's hard to see which steps are redundant or combinable to reduce complexity.

To avoid this, we'll aggressively adopt local workflows. Runnable code is easier to iterate on and keep correct through test automation.

ℹ️ BTW I've worked on projects that took more than a week to get started 😱. This was a shocking waste of time, with senior developers debugging dependencies for hours. We can and must do better.

Goals for Our Workflows

To determine our direction, let's consider the DORA research on software delivery. This research identifies what software delivery patterns lead to the best outcomes, and for this article we'll focus on two key metrics that are part of a statistically meaningful pattern that is likely to cause improvements to organizational performance:

1. Minimal time from code committed to that code running in production, ideally no more than an hour.
2. Frequent deploys, ideally each commit resulting in a deployment.

We'll align with these principles to create workflows that enable our team to continuously pull and push code changes with minimal delay.

ℹ️ BTW for more on the DORA research, check out my articles: Introduction to "Accelerate", the scientific analysis of software delivery and The Software Delivery Performance Model. Their book, Accelerate: The Science of Lean Software and DevOps, is highly recommended.

Rethinking Branches

This research leads us to a choice: To achieve high-frequency changes, branches are not ideal. Why? Because pushing commits anywhere other than main introduces latency. If we're serious about an ideal workflow, we shouldn't use branches.

For some, this is a shocking statement. How else can changes land safely? If you rely on branches read on for more details, I promise it's possible to do away with them.

ℹ️ BTW for more on trunk-based development, see the DORA research and my Beginners Intro to Trunk Based Development.

Let's start experimenting!

Bootstrapping

The most extreme onboarding will be just a single command with no prior dependencies or requirements. Here’s the simplest way to run a remote script (on Mac and Linux):

$ curl -Ssf https://…/script | sh

Imagine an onboarding guide that's just that one line 😍.

But one small constraint: We'll need to inspect the user's configuration (e.g. to check if pkgx is installed), and to do that we can't pipe to sh because that spawns a new shell. Instead, we need to source the script. And because we can't pipe to source we need to slightly change our ideal invocation:

$ curl -fsSL https://…/script > /tmp/script && source /tmp/script

But that's fine, this is still promising to be an extremely simple onboarding one-liner.

Be Careful with System Dependencies

A word of caution before we get to coding: installing system dependencies affects the user's computer as a whole, and it's not wise to try to fully automate their installation:

1. It's invasive: Some developers have strong preferences, and our script could disrupt their setup. We're asking them to trust a script they don't know, so we should write code that can't cause damage.

2. It's brittle: We can't account for everyone's different system setups, so the more sophisticated our solutions are the more we risk our code will fail.

3. It's unmaintainable: We invite pointless sophistication where developers with different preferences will add their choice to the automation, and we end up with a mess of sophistication to maintain.

And for what? pkgx already has a slick installation process, so no amount of automation is saving much time! Let's instead just identify missing dependencies and let the user handle the installation.

Maximize Trust

Let's make it clear that our script only suggests actions:

$ URL="https://raw.githubusercontent.com/gaggle/perfect-elixir/main/bootstrap"
$ curl -fsSL $URL > /tmp/bootstrap && source /tmp/bootstrap
This script bootstraps our development environment by suggesting
what dependencies need to be installed and configured.

To be clear: This script never changes or affects your system, 
it only ever inspects and makes suggestions.

Ok to proceed? [y/n]:

🖥️ Terminal

That should establish trust right from the start.

ℹ️ BTW I'm not showing bootstrap code here because it's mostly a simple script that outputs the above text. But if you'd like to follow the details you're welcome to inspect the full bootstrap script here.

The First Step

Let's first check if the developer has pkgx installed by verifying the exit code of which pkgx. If not installed, instruct the user:

Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x

User action required: Install pkgx
──────────────────────────────────
You need to install pkgx. Source this script again afterwards.

pkgx can be installed in various ways, depending on your 
preferences:

• Via Homebrew:
    $ brew install pkgxdev/made/pkgx

• Via cURL:
    $ curl -Ssf https://pkgx.sh | sh

For other ways to install see: 
  https://docs.pkgx.sh/run-anywhere/terminals

pkgx is the package manager that handles system dependencies, 
and it is not currently installed. The installation is simple,
and via Homebrew does not require sudo or other forms of 
elevated permissions.

Read more about pkgx on https://pkgx.sh

Source this script again after pkgx has been installed.

🖥️ Terminal

This way the user can proceed at their own pace, but are offered easy copy-pasteable choices.

All The Steps

To complete onboarding we'll do three more requirements:

1. Verify pkgx's shell integration.
2. Ensure the user has cloned our repository.
3. Confirm pkgx provides its developer environment (e.g., Elixir, Erlang).

Skipping details for brevity, the final bootstrapping script ends up running like this:

Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is installed ✓
• Checking pkgx shell integration… ✓
• Shell integration is active ✓
• Checking repository is cloned… ✓
• Repository is available ✓
• Checking development environment is active… ✓
• Development environment is active ✓

Good to go

Bootstrapping is done:
✓ pkgx is installed
✓ pkgx shell integration is active
✓ The repository is cloned and ready
✓ All system dependencies are available

This system has been bootstrapped and can now hook into our project 🎉

• Run this command to continue onboarding:

    $ bin/doctor

🖥️ Terminal

Here is the full flow from factory-reset machine to ready to work on the project:

And with that, we’ve unlocked a simple onboarding solution. And the simplicity of the code should invite incremental improvements by the whole team. Nice!

Now let's explore the daily development workflow scripts developers will use regularly.

ℹ️ BTW this bootstrapping script may not suit enterprise requirements but can be extended to cover various cases. Keep in mind after its initial "pkgx is installed" check the full pkgx ecosystem is available, enabling powerful tools like GitHub CLI and entire programming languages. Bootstrapping can evolve significantly based on needs!

Daily Workflows

To enable developers to quickly pull and push code changes, we need to decide on the scripting language for our workflows. With pkgx, we can use any language, but should we?

In Defence of Shell Scripting

Shell scripts are the industry standard for scripting, and are widely used and understood by almost everyone. While they may not be the most elegant choice, they are practical and often low-maintenance. We don't earn any money from writing workflow scripts, so probably our best choice is to avoid unnecessary complexities and go with what is most simple: shell scripting.

Doctor

Our first workflow component will be a script to keep our development environment up-to-date, ensuring vital preconditions are met (e.g., the local database is running, migrations are applied, Mix dependencies installed, etc.).

ℹ️ BTW I've gotten used to calling this script doctor because it verifies the health of our environment. You can choose whatever name you feel is most fitting.

First, let's catch if the user has fallen out of the pkgx ecosystem. By overlapping with where bootstrap left off we provide a fallback for unforeseen errors:

$ cat bin/doctor
#!/usr/bin/env bash
set -euo pipefail
command which pkgx && which erl && which elixir ||\
(echo "Missing system dependencies, \
run 'source bin/bootstrap'" && exit 1)

We can simulate an issue by turning off the development environment:

$ dev off
env -erlang.org=26.2.4 -elixir-lang.org=1.16.2 -postgresql.org=15.2.0 

$ bin/doctor
/usr/local/bin/pkgx
Missing system dependencies, run 'source bin/bootstrap'

$ echo $?
1

Re-enabling the environment makes the check pass:

$ dev on
env +erlang.org=26.2.4 +elixir-lang.org=1.16.2 +postgresql.org=15.2.0 

$ bin/doctor
/usr/local/bin/pkgx
/Users/cloud/.pkgx/erlang.org/v26.2.4/bin/erl
/Users/cloud/.pkgx/elixir-lang.org/v1.16.2/bin/elixir

$ echo $?
0

This directs users back to bootstrapping if the pkgx system isn't activated, quite nice.

This implementation is pretty noisy though, and mixes low-level shell implementation with high-level goals. We can improve on that by introducing an abstraction layer via a shell helper function called check:

$ cat bin/doctor
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
check "Check system dependencies" \
  "command which pkgx && which erl && which elixir" \
  "source bin/bootstrap"

$ bin/doctor
• Check system dependencies ✓

🖥️ Terminal

It's worth taking care of our terminal output to not go blind to all the mindless muck we can otherwise end up printing.

ℹ️ BTW the .shhelpers code is not directly relevant to this article, but you can find the full script here if you'd like. They're inspired by workflows introduced to me by Eric Saxby and Erik Hanson of Synchronal.

Let's check if our local database is running next:

$ git-nice-diff -U0 .
/bin/doctor
L#7:
+check "Check PostgreSQL server is running" \
+ "pgrep -f bin/postgres" \
+ "bin/db start"

$ bin/doctor
• Check system dependencies ✓
• Check PostgreSQL server is running x
> Executed: pgrep -f bin/postgres
Suggested remedy: bin/db start
(Copied to clipboard)

$ bin/db start
• Creating /Users/cloud/perfect-elixir/priv/db ✓
• Initializing database ✓
• Database started:
waiting for server to start.... done
server started
↳ Database started ✓

$ bin/doctor
• Check system dependencies ✓
• Check PostgreSQL server is running ✓

🖥️ Terminal

The doctor pattern is now clear: Check for a condition, suggest a fix. Simple to extend, easy to understand.

ℹ️ BTW the bin/db script abstracts logic away from the doctor script and provides a handy way for developers to manage their database. Its implementation isn't directly relevant but you can read the full script here.

Let's skip to having added all necessary checks for our app to start:

$ bin/doctor
Running checks…

• Check system dependencies ✓
• Check developer environment ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready

🖥️ Terminal

And now we can start our app 🎉:

$ iex -S mix phx.server
[info] Running MyAppWeb.Endpoint with Bandit 1.4.2 at 127.0.0.1:4000 (http)
[info] Access MyAppWeb.Endpoint at http://localhost:4000
[watch] build finished, watching for changes...
Erlang/OTP 26 [erts-14.2.4] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [dtrace]

Interactive Elixir (1.16.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

🖥️ Terminal

We now have bin/doctor ensuring our system is ready. And new developers can go from a factory-reset machine to having our product running locally in just a handful of minutes by running bootstrap & doctor.

Update

Next, we'll create a script to easily get the latest code. This will be a replacement for git pull, as it will pull down the latest changes and run necessary commands to apply changes correctly.

ℹ️ BTW especially teams that use trunk-based development can generate several dozens of commits per day, so there's good need for a script like this.

First, let's run git pull, and then ensure mix dependencies are up-to-date and compiled. The .shhelpers library from before has a step function that runs a command but hides the output unless an error occurs, which is perfect for this:

$ cat bin/update
#!/usr/bin/env bash
set -euo pipefail
source "$(dirname "$0")/.shhelpers"
check "Check branch is main" \
  '[ "$(git rev-parse --abbrev-ref HEAD)" = "main" ]' \
  "git checkout main"
step "Pulling latest code" \
  "git pull origin main --rebase"
step "Installing dependencies" "mix deps.get"
step "Compiling dependencies" "mix deps.compile"
bin/doctor

$ bin/update
• Check branch is main ✓
• Pulling latest code ✓
• Installing dependencies ✓
• Compiling dependencies ✓
Running checks…
• Check system dependencies ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready

🖥️ Terminal

This script makes it easy to integrate the latest changes, and because it ends with running doctor we're constantly ensuring our system is in a good state. And we'll add more steps to this script as needed whenever we discover additional tasks that should be run after pulling new code.

ℹ️ BTW usually update would also apply migrations, but we don't have any yet so I've skipped that for now.

Shipit

The final workflow script, shipit, is crucial because it will let us safely ship changes. It must ensure our code is in a shippable state by running test automation and other quality gates before pushing the code.

Our needs are simple right now as we don't have much code: we just need to run unit-tests and formatting checks. Here's how we can do that:

$ cat bin/shipit
#!/usr/bin/env bash
set -euo pipefail

source "$(dirname "$0")/.shhelpers"

bin/update
step --with-output "Run tests" "mix test"
check "Check files are formatted" "mix format --check-formatted" "mix format"
step "Pushing changes to main" "git push origin main"
cecho "\n" -bB --green "✓" --green " Shipped! 🚢💨"

And the result is:

$ bin/shipit
Integrating changes…
• Check active branch ✓
• Pulling latest code ✓
• Installing dependencies ✓
• Compiling dependencies ✓

Running checks…
• Check system dependencies ✓
• Check PostgreSQL server is running ✓
• Check PostgreSQL server has required user ✓
• Check mix hex ✓
• Check mix dependencies ✓
• Check PostgreSQL database exists ✓

✓ System is healthy & ready

Checking code…
• Run tests:
.....
Finished in 0.1 seconds (0.05s async, 0.06s sync)
5 tests, 0 failures

Randomized with seed 297141
↳ Run tests ✓
• Check files are formatted ✓
• Pushing changes to main ✓

✓ Shipped! 🚢💨

🖥️ Terminal

This provides a safe and quick way to ship code by first updating the code to ensure the environment is in sync, then running tests and checking for any issues, and finally pushing to main. This workflow maximizes Continuous Integration (CI) and Continuous Delivery (CD) by constantly integrating changes and pushing code to production with minimal latency.

All that's left now is to practice shipping frequently, and continuously engage customers for feedback!

ℹ️ BTW it's beneficial to adopt these scripts while they're still raw and simple. Waiting for them to be "perfect" is IMO a mistake because the clarity and ease of iteration of the initial versions is what builds trust in the workflows. The initial scripts should cover essential needs, and then be allowed to naturally expand. This engages the team and maximizes collective involvement.

Check All the Things

We've established workflows that let us continuously integrate changes with bin/update and push changes with bin/shipit (replacing git pull and git push). While these scripts can be improved and made more robust by adding more quality gates (e.g., run Dialyzer, prevent compiler warnings, run security scans), there's one aspect we can't automate: The review.

Code is improved by multiple pairs of eyes, but how can that be done without adding branches and latency? The answer is simple yet impactful:

Reviewing must also happen locally.

Some developers may resist this idea, but it aligns with modern development practices: Discrete reviews, often from code-changes that have been hidden away in branches for hours and days, add latency and friction. We should instead aim for continuous code reviewing.

So when a commit is ready, get it reviewed immediately. Don't wait, don't delay, and don't start other work until the current work is reviewed. And to further reduce disruptions just code it together: share a workstation (or use screen sharing remotely) and develop the code collaboratively. This way, changes flow to main without obstacles, enabling true continuous integration and continuous delivery.

Then, practice taking many more much smaller steps, shipping dozens of times an hour.

Now we're achieving real continuous integration and continuous delivery 🤩.

ℹ️ BTW there is extensive literature on pair and whole-team programming. While negative pairing can be exhausting, positive pairing is very enjoyable 😊. Articles like Pair Programming by Martin Fowler explain the dos and don'ts, and Dave Farley's videos explore the topic insightfully. Additionally, Woody Zuill's Mob Programming: A Whole Team Approach offers insights beyond pairing. For continuous improvement, Many More Much Smaller Steps by GeePaw Hill provides excellent inspiration.

Test Automating Our Workflows

Testing scripts is often considered too hard, but without tests, iterating on our scripts becomes increasingly difficult. Let's tackle this challenge step-by-step.

Testability

To make scripts testable, we need to mock external calls. Running the script in a real environment for every test isn't feasible, so we need a way to simulate these calls.

Mocking System Calls

We'll use a call command that by default simply wraps external calls, but it also allows itself to be mocked during tests. Here's a basic implementation we can wire into the start of our scripts, which defines call only if it doesn't already exist:

# Define call if 'call' is not already a command
if ! type call >/dev/null 2>&1; then
  call() { "$@"; }
fi

Under test conditions, we can replace call with a mock implementation:

$ call() { echo "called with: $*" > call.log; return 1; }
$ source bin/bootstrap
…
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x
…

$ cat call.log
called with: which pkgx

What we see above is that our mocked call gets invoked when we run bootstrap, allowing us to control its behavior and verify that our script responds correctly.

To put it into our bootstrapping context, we can wrap it all into an easier-to-use script that lets us configure and create a mocked call command, like this:

$ export MOCK=$(test/mcall configure "which pkgx|1|")
$ source test/mcall
$ source bin/bootstrap
…
Ok to proceed? [y/n]: y
• Checking for pkgx… ✓
• pkgx is not installed x
…
$ test/mcall assert
All mocks used ✓

And that's it. That's one fully tested use case right there!

Well, sort of: I did have to manually answer y to bootstrap's prompt. How do we automate that?

ℹ️ BTW it's possible this complexity in testing shell scripts should be a reason to switch away from shell scripting. Maybe if we were writing all this in a more comprehensive programming language, we could use their mature test-runners to more easily achieve test-automation? It's worth considering, although for now I'll stay the course to see if this is even possible to solve.

Expect

Next, we need to interact with our scripts and assert their outputs. For this, we'll use expect, a tool for automated interaction with programs:

$ expect -c 'spawn echo "foo"; expect "foo"; expect eof'
spawn echo foo
foo

To test our bootstrap script, we can create an expect script:

$ cat test/expect.exp
#!/usr/bin/env expect
set timeout 3
expect_before {
    timeout { puts "timeout"; exit 2 }
}
spawn bash
send "source bin/bootstrap\r"
expect "Ok to proceed? [y/n]:"
send "n\r"
send "exit\r"
expect eof

Running this script simulates user interaction and validates the output:

$ ./test/expect.exp
…
Ok to proceed? [y/n]: n
exit
bash-5.2$ exit
exit

$ echo $?
0

Cool.

And note: expect is quite extensible because it's based on the Tcl language (pronounced "tickle"). It has a great history dating back to the late 80s, and is a proper language with procedures and conditionals and much more.

ℹ️ BTW the full expect script I ended up with takes additional arguments and pretty-prints some details. It's available here if you're curious.

🖥️ Terminal

bats! 🦇

To manage our tests, ensuring they all get run and tracking which ones pass and which fail, we'll turn to the Bash Automated Testing System (bats). It lets us define tests in a bootstrap.bats file like this:

@test "pkgx not installed" {
  run_mocked_scenario '
which pkgx|1|pkgx not found
' \ '
send "source bin/bootstrap\n"
exp "Ok to proceed?"
send "y\n"
exp "pkgx is not installed"
exp "User action required: Install pkgx"
exp_prompt
'
}

$ bats test/bootstrap.bats
bootstrap.bats
 ✓ pkgx not installed

1 test, 0 failures

ℹ️ BTW here I'm skipping over some details to not extend this article even more. The above snippet actually calls a bats helper function that orchestrates mcall and expect. The full bootstrap.bats is available here if you'd like to follow all the details.

Finally, we can write tests for all possible use cases to ensure our script behaves as expected:

$ bats test/bootstrap.bats
bootstrap.bats
 ✓ no to proceed
 ✓ pkgx not installed
 ✓ pkgx is too old
 ✓ pkgx is missing dev integration
 ✓ pkgx is missing env integration
 ✓ folder is not a repository
 ✓ remote is not expected repository
 ✓ no erl so should activate dev
 ✓ no elixir so should activate dev
 ✓ no psql so should activate dev
 ✓ good to go with git remote
 ✓ good to go with https remote

12 tests, 0 failures

🖥️ Terminal

These tests help guard against regressions and ensure our scripts work across different shells. They've been very helpful in driving out several small bugs that would have otherwise plagued these scripts.

If you have simpler methods for testing interactive scripts, please share them!

Conclusion

We've covered a lot today, and have come away with a set of workflow scripts that streamline onboarding and daily development tasks. These scripts support rapid code iteration and align with best scientific practices, and avoid relying on latency-adding workflows such as branches and pull-requests. I think they will form a great foundation for many projects, to foster a culture of efficiency, quality, and continuous improvement.