Add context to errors

It is Friday evening. Bob just wants to wrap up the small bug fix in the service developed by his team, be done for the week, and pack up for his weekend trip.

Bob is tired but determined. He quickly glances through the code, it is nice and readable, so with a quick search, and a few jumps to definition, he finds the suspect. He shoots a few lines of code before asking any questions, and... Done!

With good old ls Bob identifies the tests directory, and since it is a Rust project he knows it will contain integration tests. His fingers dance on the keyboard again as he quickly adds the test case, balancing on the edge of razor-sharp focus and exhaustion.

He formats the code, slams ctrl + s, opens the terminal, and types cargo test before the cursor has a chance to blink once. He can already see all the green oks, his mind can no longer stand against the gravity of the weekend. Code compiles painfully slowly, as he stretches his fingers for the last (git) push. There are no errors, not even a warning, tests start-up and...

thread 'test_whatever' panicked at Os { code: 2, kind: NotFound, message: "No such file or directory" }', tests/source_code.rs:291:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Bob scratches his head once, and again a bit harder... He takes a deep breath and blows off fallen hair from his keyboard while he exhales slowly, shrinking in his chair like a deflated balloon. A look at a clock reveals 8 pm. Bob stares at the code, but he does not see much, while all his brain cells converge around trying to answer a single, important question: "What f***** file?"

Convenient does not mean good

Since I started with the Rust example let me continue on this path for now, however, the problem is not language-specific.

As with everything Rust gives many options when it comes to dealing with errors. Most notably we can simply unwrap() causing the thread to panic or return from the function early when an error occurs with the ? operator (of course there are some constraints to it about which you can read in the documentation).

While unwrap()ing most often is used with caution (rightfully so), the ? operator is almost as convenient especially when paired with a dynamic error object in the form of Box<dyn std::error::Error>, and given how clean it makes the code look its use is ubiquitous.

When it comes to test code both practices are fine, at least in the majority of cases. While in the "surface code" unwraping or returning Box<dyn std::error::Error> early with ? is a good way to keep it concise, the problem begins -- as usual -- when we add more layers.

Let's consider a very simple test for a single function:

#[test]
fn test_do_something() {
    let out = do_something().unwrap();
    assert_eq!(&out, "something");
}

If our do_something function fails, we can find it out very easily, since as
part of the error we directly get the line of code that unwrapped:

---- tests::test_do_something stdout ----
thread 'tests::test_do_something' panicked at 'called `Result::unwrap()` on an `Err` value: Something went wrong', src/main.rs:15:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If the test is lengthy and more complex, a good practice in my experience would be to still add some more information on what exactly failed. A month from now, you will not remember what line 344 does, you will have to navigate there, read some code, and load up your mind with long discarded context. To smooth those experiences out expect is your friend:

#[test]
fn test_do_something() {
    let out = do_something()
        .expect("failed to do something");
    assert_eq!(&out, "something");
}

---- tests::test_do_something stdout ----
thread 'tests::test_do_something' panicked at 'failed to do something: Something went wrong', src/main.rs:16:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Code bases, however, tend to grow and become more complicated, there are more moving pieces and interdependencies, and to fight the growing probability of errors creeping into cracks between our modules and their interactions we often reach for integration tests. Those are rarely that simple.

It is there, where we are going to have setup code, perhaps spinning up a database, creating temporary files, populating fake data to test filtering logic, or just having a lot of helper functions.

And here we come back to Bob's story.

Let's consider another example. This time we have a bunch of setup code and some functions are used in multiple places. In a real-world project this test together with all the helpers could have hundreds of lines of code:

type Result<T> = std::result::Result<T, Box<dyn std::error::Error>>;

fn setup_test() -> Result<()> {
    initial_setup()?;
    populate_data()?;
    let something = do_something()?;
    Ok(())
}

fn initial_setup() -> Result<()> {
    let something = do_something()?;
    // Some initial setup code
    Ok(())
}

fn populate_data() -> Result<()> {
    let something = do_something()?;
    // Some more setup code
    Ok(())
}

fn do_something() -> Result<String> {
    Err(String::from("Something went wrong").into())
}

#[test]
fn test_do_something() {
    setup_test().unwrap();
    // Test logic...
}

---- tests::test_do_something stdout ----
thread 'tests::test_do_something' panicked at 'called `Result::unwrap()` on an `Err` value: "Something went wrong"', src/main.rs:34:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Now, we can only suspect that do_something failed somewhere, and it holds true only if we know that this particular error Something went wrong comes only from the do_something function. Imagine that instead, it is some standard library error such as No such file or directory, we would find ourselves in Bob's shoes scratching hair out from our scalp. Not even the RUST_BACKTRACE=full would come to our rescue.

Context for context

To make Bob's life (and likely their own) easier, people a long time ago figured out that adding context to errors at different layers makes them much easier to debug, and in general understand what happens in the application. This article is by no means any kind of tutorial and is more of a rant, but I feel obligated to provide at least a basic example of how to achieve it.

Since I do not cover the basics of Rust error handling here if you are new to Rust you can have a look at great talk by Tim McNamara that covers the error handling journey.

With Rust, the easiest way is probably to use anyhow crate that allows us to call .context(...) on any Result type where Err part implements std::error::Error trait, and turn it into anyhow::Result (or Result<T, anyhow::Error>) wrapping the error with any additional information.

Using anyhow we can transform the last example:

use anyhow::Context;

fn setup_test() -> anyhow::Result<()> {
    initial_setup()
        .context("failed to do initial setup")?;
    populate_data()
        .context("failed to populate data")?;
    let something = do_something()
        .context("failed to do something during setup")?;
    Ok(())
}

fn initial_setup() -> anyhow::Result<()> {
    let something = do_something()?;
    // Some initial setup code
    Ok(())
}

fn populate_data() -> anyhow::Result<()> {
    let something = do_something()?;
    // Some more setup code
    Ok(())
}

fn do_something() -> anyhow::Result<String> {
    Err(anyhow::Error::msg("Something went wrong"))
}

#[test]
fn test_do_something() {
    setup_test().expect("failed to setup test");
    // Test logic...
}

and have a much saner error message to work with:

---- tests::test_do_something stdout ----
thread 'tests::test_do_something' panicked at 'failed to setup test: failed to do initial setup

Caused by:
    Something went wrong', src/main.rs:39:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

While very convenient and straightforward, it may not be the solution for all cases, for example when we want to retain a custom error type. Sadly, for this scenario, I am yet to find a low-friction way to eat a cookie and have a cookie. Some reasonable practices I have found would be:

• Introduce an error type for each module (or layer of abstraction) and extend them appropriately on different levels (leveraging the source method of the std::error::Error trait). It is not the most convenient approach and requires quite a bit of "boring code", however, it might be the most precise one. Crates like thiserror can help a lot here reducing the amount of boilerplate code.
• With anyhow we can relatively easily downcast the error, so it may be a suitable solution in certain cases.

• Wrapping the Result in Result, in order to have a way to represent a specific application error and all other errors that do not need special handling (here you can use anyhow again):
  

  pub enum MyAppError {
      ...
  }
  
  pub fn do_something() -> anyhow::Result<std::result::Result<(), MyAppError>>
  

  In this case, if we encounter some unexpected IO error we can add context to it with anyhow, but if there is an application-specific error, that we want the calling code to handle we would return Ok(Err(MyAppError {...})).

For more practical resources and deep dives rather than philosophical ranting, you can take a look at:

• Jane Lusby's talk about error handling.
• Luca Palmieri's talk on error handling with an example in a web application
• matklad's Study of std::io::Error

It (obviously) is not just about tests

While I started my rant from the test code, I later realized that the value of adding context to errors may not be as obvious to everyone as it is to me. Then I remembered that too many times in my life I have seen this:

err := doSomething()
if err != nil {
    return err
}

and it is not a rant about Go's error handling, but cryptic errors during development or even worse production. Bear in mind that it is often other people who will troubleshoot your code and the error on line 537 of magic.go won't tell them anything. To spare those SREs a few swear words we often do not even need to add a line of code...

err := doSomething()
if err != nil {
    return fmt.Errorf("failed to do something: %w", err)
}

Say what you want about Go error handling, but its simplicity however verbose is actually great in practice. While it definitely lacks the precision of Rust type system, you just wrap the error and spare yourself hours of overthinking what is the best approach for error handling in my application as well as fighting the conveniet temptation of simply ? it away to the caller. But, that is a rant for another day...

Conclusion: helpful error messages are helpful

From my experience of troubleshooting other people's software (as well as my own ¯\_(ツ)_/¯), it is better to err on the side of more context than less. While I am not suggesting writing an essay in every error context, or even that, we need to add it in every possible place, I will take a bit redundant error such as:

failed to load config: failed to read config file: failed to parse config from JSON: invalid character 'a' looking for the beginning of object key string

over simply:

invalid character 'a' looking for the beginning of the object key string

We certainly want to strive to keep errors concise, however, from two extremes I chose the former.

Thanks for sticking with me on this rant. The whole point here is that we all could be more mindful about using things like the ? operator, if err != nil { return err }, and any other ways of just pushing errors up the call stack. Look at error messages while you writing or reviewing the code and ask yourself "will I or someone who never touched this code get at least a sense of the problem seeing this error message?".