Like the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press.

Exceptions.

One of the arch-nemeses of many a programmer. In many languages, we're trained to associate an exception with some degree of failure; something, somewhere, was used improperly.

What if I told you that you don't have to be afraid of exceptions? That they wanted to be your friend and help you write better code?

Python offers many familiar error handling tools, but the way we use them may look quite different from what you're used to, and it can help you do quite a bit more than just cleaning up messes. You might even say, error handling in Python is bigger on the inside.

Geronimo!

Playing Catch

Just in case exceptions are unfamiliar to you, let's start with the general definition...

exception: (computing) An interruption in normal processing, typically caused by an error condition, that can be handled by another part of the program. (Wiktionary)

Let's look at a simple example for starters:

def initiate_security_protocol(code):
    if code == 1:
        print("Returning onboard companion to home location...")
    if code == 712:
        print("Dematerializing to preset location...")

code = int(input("Enter security protocol code: "))
initiate_security_protocol(code)

>>> Enter security protocol code: 712
Dematerializing to preset location...
>>> Enter security protocol code: seven one two
Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/security_protocols.py", line 7, in <module>
    code = int(input("Enter security protocol code: "))
ValueError: invalid literal for int() with base 10: 'seven one two'

Clearly, this a problem. We don't want our program to abruptly crash because the user entered something weird. As the joke goes...

A QA Engineer walks into a bar. He orders a beer. He orders five beers. He orders -1 beers. He orders a lizard.

We want to protect against weird input. In this case, there's only one significant failure point: that int() function. It expects to receive something it can cast to an integer, and if it doesn't get it, it raises a ValueError exception. To handle this properly, we wrap the code that might fail in a try...except block.

try:
    code = int(input("Enter security protocol code: "))
except ValueError:
    code = 0
initiate_security_protocol(code)

When we test our code again, we won't get that failure. If we couldn't get the information we needed from the user, we'll just use the code 0 instead. Naturally, we can rewrite our initiate_security_protocol() function to handle a code of 0 differently, although I won't show that here, just to save time.

Gotcha Alert: For whatever reason, as a multi-language programmer, I often forget to use except in Python, instead of the catch statement most other languages use. I've literally mistyped it three times in this article already (and then immediately fixed it.) This is just a point of memorization. Thankfully, Python has no catch keyword, so that makes syntax errors a LOT more obvious. If you know multiple languages, when you get these confused, don't panic. It's except, not catch.

Reading Traceback

Before we dive into some of the deeper details of the try...except statement, let's look back at that error statement again. After all, what good is an article about error handling if we don't talk about error messages? In Python, we call this a Traceback, because it traces the origins of the error from the first line of code involved to the last. In many other languages, this would be referred to as a stack trace.

Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/security_protocols.py", line 7, in <module>
    code = int(input("Enter security protocol code: "))
ValueError: invalid literal for int() with base 10: 'seven one two'

I have the habit of reading these messages bottom-to-top, because it helps me get to the most important information first. If you look at the last line, you see ValueError, which is the particular exception that has been raised. The exact details follow; in this case, it wasn't possible to convert the string 'seven one two' to an integer with int(). We also learn that it's attempting to convert to a base 10 integer, which is potentially useful information in other scenarios. Imagine, for example, if that line said...

ValueError: invalid literal for int() with base 10: '5bff'

That's perfectly possible if we forget to specify base-16, as in int('5bff', 16), instead of the default (base 10). In short, you should always thoroughly read and understand the last line of the error message! There have been too many times where I've half-read the message, and spent half an hour chasing the wrong bug, only to dicover I forgot a parameter or used the wrong function.

Above the error message is the line of code that the error came from (code = int(input("Enter security protocol code: "))). Above that is the absolute path to the file (security_protocols.py) and the line number 7. The statement in <module> means the code is outside of any function. In this example, there's only one step in the callback, so let's look at something slightly more complicated. I've changed and expanded the code from earlier.

Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/databank.py", line 6, in <module>
    decode_message("Bad Wolf")
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/databank.py", line 4, in decode_message
    initiate_security_protocol(message)
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/security_protocols.py", line 2, in initiate_security_protocol
    code = int(code)
ValueError: invalid literal for int() with base 10: 'Bad Wolf'

We're getting a similar error as before - we're trying to convert a string into an integer, and it's not working. The second-to-last line shows us the code that's failing; sure enough, there's the call to int() that's raising this. According to the line above that, this problematic code is at line 2 of security_protocols.py, inside of the initiate_security_protocol() function. Great! We could actually stop there and go wrap it in a try...except. See why reading bottom-to-top saves time?

However, let's imagine it's not that simple. Maybe we don't have the option to modify security_protocols.py, so we need to prevent the problem before that module is executed. If we look at the next pair of lines up, we see that on databank.py line 4, inside the decode_message() function, we're calling the initiate_security_protocol() function that is having the problem. That function in turn is being called on line 6 of databank.py, outside of any function, and that's where we're passing the argument "Bad Wolf" to it.

The data input isn't the problem, since we want to decode the message "Bad Wolf." But, why are we passing a message we're trying to decode right to the security protocols? Perhaps we need to rewrite that function instead (or in addition to the other change?). As you can see, the Traceback is incredibly important in understanding where errors originate from. Make a habit of reading it thoroughly; many useful bits of information can hide in unexpected places.

By the way, that first line is the same every time, but it is very useful if you forget how to read these messages. The most recently executed code is listed last. Thus, as I've said before, you should read them from the bottom up.

Your Friend, the Exception

"It's easier to ask forgiveness than it is to get permission." -Rear Admiral Grace Hopper

This quote was originally about taking initiative; if you believe in an idea, take a risk on it instead of waiting for permission from someone else to pursue it. In this case, however, it's an excellent description of Python's philosophy of error handling: if something could regularly fail in one or more specific ways, it is often best to use a try...except statement to handle those situations.

This philosophy is formally named "Easier to Ask Forgiveness than Permission", or EAFP.

That's a bit abstract, so let's consider another example. Let's say we want to be able to look up information in a dictionary.

datafile_index = {
    # Omitted for brevity.
    # Just assume there's a lot of data in here.
}

def get_datafile_id(subject):
    id = datafile_index[subject]
    print(f"See datafile {id}.")

get_datafile_id("Clara Oswald")
get_datafile_id("Ashildir")

See datafile 6035215751266852927.

Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/databank.py", line 30, in <module>
    get_datafile_id("Ashildir")
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/databank.py", line 26, in get_datafile_id
    id = datafile_index[subject]
KeyError: 'Ashildir'

The first function call works just fine. We search the dictionary database_index for the key "Clara Oswald", which exists, so we return the value associated with it (6035215751266852927), and print that data out in our lovely little formatted print() statement. The second function call, however, fails. The exception KeyError is raised, because"Ashildir" isn't a key in the dictionary.

Technical Note: Python offers collections.defaultdict as another solution to this exact problem; attempting to access a key that doesn't exist will create the key/value pair in the dictionary, using some default value. However, since this is an example for demonstrating error handling, I'm not using it.

Since we can't reasonably be expected to know or memorize all the keys in the dictionary, especially in a real-world scenario, we need some way of handling the common situation of trying to access a key that doesn't exist. Your first instinct might be to check for the dictionary key before attempting to access it...

def get_datafile_id(subject):
    if subject in datafile_index:
        id = datafile_index[subject]
        print(f"See datafile {id}.")
    else:
        print(f"Datafile not found on {subject})

In Python culture, this approach is called "Look Before You Leap" [LBYL].

But this isn't the most efficient way! The "forgiveness, not permission" comes into play here: instead of testing first, we use try...except.

def get_datafile_id(subject):
    try:
        id = datafile_index[subject]
        print(f"See datafile {id}.")
    except KeyError:
        print(f"Datafile not found on {subject}")

The logic behind this is simple: instead of accessing the key twice (the "permission" method), we only access it once, and use the actual exception as the means of branching our logic.

In Python, we don't consider exceptions to be something to be avoided. In fact, try...except is a regular part of many Python design patterns and algorithms. Don't be afraid of raising and catching exceptions! In fact, even keyboard interruptions are handled this way, via the KeyboardInterrupt exception.

Gotcha Alert: try...except is a powerful tool, but it isn't for everything. For example, returning None from a function is often considered better than raising an exception. Only raise an exception when an actual error occurs that is best handled by the caller.

Beware the Diaper Anti-Pattern

Sooner or later, every Python developer discovers this works:

try:
    someScaryFunction()
except:
    print("An error occured. Moving on!")

A bare except allows you to catch all exceptions in one. In his book "How To Make Mistakes in Python" [O'Reilly, 2018], Mike Pirnat calls this the diaper pattern, and it is a really, really bad idea. I'll allow him to summarize...

...all the precious context for the actual error is being trapped in the diaper, never to see the light of day or the inside of your issue tracker. When the “blowout” exception occurs later on, the stack trace points to the location where the secondary error happened, not to the actual failure inside the try block.

Long story short, you should always explicitly catch a particular exception type. Any failure that you cannot forsee probably has relation to some bug that needs to be resolved; for example, when your super complicated search function suddenly starts raising an OSError instead of the expected KeyError or TypeError.

As usual, the Zen of Python has something to say about this...

Errors should never pass silently.
Unless explicitly silenced.

To put that yet another way, this ain't Pokemon - you shouldn't catch 'em all!

You can read more about why the diaper pattern is such a terrible idea in detail in the article The Most Diabolical Python Antipattern.

Except, Else, Finally

Great, so I don't just catch all the exceptions in one fell swoop. So, how do I handle multiple possible failures?

You'll be glad to know that Python's try...except has a lot more tools than it first shows.

class SonicScrewdriver:

    def __init__(self):
        self.memory = 0

    def perform_division(self, lhs, rhs):
        try:
            result = float(lhs)/float(rhs)
        except ZeroDivisionError:
            print("Wibbly wobbly, timey wimey.")
            result = "Infinity"
        except (ValueError, UnicodeError):
            print("Oy! Don't diss the sonic!")
            result = "Cannot Calculate"
        else:
            self.memory = result
        finally:
            print(f"Calculation Result: {result}\n")


sonic = SonicScrewdriver()

sonic.perform_division(8, 4)
sonic.perform_division(4, 0)
sonic.perform_division(4, "zero")

print(f"Memory Is: {sonic.memory}")

Before I show you the output, take a careful look at the code. What do you think each of the three sonic.perform_division() function calls will print out? What's ultimately stored in sonic.memory? See if you can figure it out.

Think you've got it? Let's see if you're right.

Calculation Result: 2.0

Wibbly wobbly, timey wimey.
Calculation Result: Infinity

Oy! Don't diss the sonic!
Calculation Result: Cannot Calculate

Memory Is: 2.0

Were you suprised, or did you get it right? Let's break that down.

try: is, of course, the code we're attempting to run, which may or may not raise an exception.

except ZeroDivisionError: occurs when we try to divide by zero. We say the value "Infinity" is the result of the calcuation, in this case, and print out an apt message about the nature of the spacetime continuum.

except (ValueError, UnicodeError): occurs whenever one of these two exceptions is raised. ValueError happens whenever any of the arguments we passed could not be cast by float(), while UnicodeError occurs if there's a problem encoding or decoding Unicode. Actually, that second one was just included to make a point; the ValueError would be sufficient for all believable scenarios where the argument couldn't be turned into a float. In either case, we use the value "Cannot Calculate" as our result, and remind the user not to make unreasonably demands of the hardware.

Here's where things get interesting. else: runs only if no exception was raised. In this case, if we had a valid numeric result of our division calculation, we actually want to store that in memory; conversely, if we got "Infinity" or "Cannot Calculate" as our result, we do not store that.

The finally: section runs no matter what. In this case, we print out the results of our calculation.

The order does matter. We must follow the pattern try...except...else...finally. The else, if present, must come after all the except statements. The finally is always last.

It is initially easy to confuse else and finally, so be sure you understand the difference. else runs only if no exception was raised; finally runs every time.

How Final is finally?

What would you expect the following to do?

class SonicScrewdriver:

    def __init__(self):
        self.memory = 0

    def perform_division(self, lhs, rhs):
        try:
            result = float(lhs)/float(rhs)
        except ZeroDivisionError:
            print("Wibbly wobbly, timey wimey.")
            result = "Infinity"
        except (ValueError, UnicodeError):
            print("Oy! Don't diss the sonic!")
            result = "Cannot Calculate"
        else:
            self.memory = result
            return result
        finally:
            print(f"Calculation Result: {result}\n")
            result = -1


sonic = SonicScrewdriver()

print(sonic.perform_division(8, 4))

That return statement under else should be the end of things, right? Actually, no! If we run that code...

Calculation Result: 2.0

2.0

There's two important observations from this:

1. finally is running, even after our return statement. The function doesn't exit like it normally would.

2. The return statement is indeed running before the finally block executes. We know this because the result output was 2.0, not the -1 we assigned to result in our finally statement.

finally will be run every time, even if you have a return elsewhere in the try...except structure.

However, I also tested the above with an os.abort() instead of return result, in which case the finally block never ran; the program aborted outright. You can stop program execution outright anywhere, and Python will just drop what it's doing and quit. That rule is unchanged, even by the unusual finally behavior.

Being Exceptional

So, we can catch exections with try...except. But what if we actually want to throw one?

In Python terminology, we say we raise an exception, and like most things in this language, accomplishing that is obvious: just use the raise keyword:

class Tardis:

    def __init__(self):
        pass

    def camouflage(self):
        raise NotImplementedError('Chameleon circuits are stuck.')

tardis = Tardis()
tardis.camouflage()

When we execute that code, we see the exception we raised.

Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/tardis.py", line 10, in <module>
    tardis.camoflague()
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/tardis.py", line 7, in camoflague
    raise NotImplementedError('Chameleon circuits are stuck.')
NotImplementedError: Chameleon circuits are stuck.

Ah well, I guess we're stuck with that police box form. At least that makes it easier to remember where we parked.

Gotcha Alert: The NotImplementedError exception is one of the built-in exceptions in Python, sometimes used to indicate a function should not be used yet because it isn't finished (but will be someday). It's not interchangable with the NotImplemented value. See the documentation to learn when to use each.

The critical code, obviously, is raise NotImplementedError('Chameleon circuits are stuck.'). After the raise keyword, we give the name of the exception object to raise. In most cases, we create a new object from an Exception class, as you can see from the use of parenthesis. All exceptions accept a string as the first argument, for the message. Some exceptions accept or require more arguments, so see the documentation.

Using The Exception

Sometimes we need to do something with the exception after catching it. We have some very simple ways of doing this.

The most obvious would be to print the message from the exception. To do this, we'll need to be able to work with the exception object we caught. Let's change the except statement to except NotImplementedError as e:, where e is the name we're "binding" to the exception object. Then, we can use e directly as an object.

tardis = Tardis()

try:
    tardis.camouflage()
except NotImplementedError as e:
    print(e)

The exception class has defined its __str__() function to return the exception message, so if we cast it to a string (str()), that's what we'll get. You might remember from a previous article that print() automatically casts its argument to a string. When we run this code, we get...

Chameleon circuits are stuck.

Great, that's easy enough!

Bubbling Up

Now, what if we want to raise the exception again?

Wait, what? We just caught the thing. Why raise it again?

One example is if you needed to do some cleanup work behind the scenes, but still ultimately wanted the caller to have to handle the exception. Here's an example...

class Byzantium:

    def __init__(self):
        self.power = 0

    def gravity_field(self):
        if self.power <= 0:
        raise SystemError("Gravity Failing")


def grab_handle():
    pass


byzantium = Byzantium()

try:
    byzantium.gravity_field()
except SystemError:
    grab_handle()
    print("Night night")
    raise

In the example above, we simply want to grab onto something solid (grab_handle()) and print an additional message, and then let the exception go with raise. When we re-raise an exception, we say it bubbles up.

Night night
Traceback (most recent call last):
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/byzantium.py", line 18, in <module>
    byzantium.gravity_field()
  File "/home/jason/Code/FiringRanges/PyFiringRange/Sandbox/byzantium.py", line 8, in gravity_field
    raise SystemError("Gravity Failing")
SystemError: Gravity Failing

Gotcha Alert: You may think we need to say except SystemError as e: and raise e or something, but that's overkill. For an exception to bubble up, we only need to call raise by itself.

Now, what if we want to add some additional information while bubbling up the exception? Your first guess might just be to raise a new exception altogether, but that introduces some problems. To demonstrate, I'm going to add another layer to the execution order. Note, when I handle that SystemError, I'm raising a new RuntimeError instead. I'm catching that new exception in the second try...except block.

byzantium = Byzantium()

def test():
    try:
        byzantium.gravity_field()
    except SystemError:
        grab_handle()
        raise RuntimeError("Night night")

try:
    test()
except RuntimeError as e:
    print(e)
    print(e.__cause__)

When we run this, we get the following output.

Night night
None

When we catch that new exception, we have absolutely no context on what caused it. To solve this problem, Python 3 introduced explicit exception chaining in PEP 3134. Implementing it is easy. Take a look at our new test() function, which is the only part I've changed from the last example.

byzantium = Byzantium()

def test():
    try:
        byzantium.gravity_field()
    except SystemError as e:
        grab_handle()
        raise RuntimeError("Night night") from e

try:
    test()
except RuntimeError as e:
    print(e)
    print(e.__cause__)

Did you catch what I'm doing there? In the except statement, I bound the name e to the original exception we were catching. Then, when raising the new RuntimeError exception, I chained it to the previous exception with from e. Our output is now...

Night night
Gravity Failing

When we run that, our new exception remembers from whence it came - the previous exception is stored in its __cause__ attribute (printed in the second line of output). This is especially useful for logging.

There are many other tricks you can do with exception classes, especially with the introduction of PEP 3134. As usual, I recommend you read the documentation, which I link to at the end of the article.

Custom Exceptions

Python has a whole bunch of exceptions, and their uses are very well documented. I frequently refer to this list of exceptions when I'm selecting the right one for the job. Sometimes, however, we just need something as bit more...custom.

All error-type exceptions are derived from the Exception class, which is derived in turn from the BaseException class. The reason for this dual heiarchy is so you can catch all error Exceptions without also reacting to special, non-system-exiting exceptions like KeyboardInterrupt. Of course, this won't matter much to you in practice, since except Exception is practically always just another form of the Diaper Anti-Pattern I referred to earlier. And anyhow, it is not recommended that you derive directly from BaseException - just know that it exists.

When making a custom exception, you can actually derive from any exception class you like. Sometimes, it's best to derive from the exception that is closest in purpose to the one you're making. However, if you're at a loss, you can just derive from Exception.

Let's make one, shall we?

class SpacetimeError(Exception):
    def __init__(self, message):
        super().__init__(message)

class Tardis():

    def __init__(self):
        self._destination = ""
        self._timestream = []

    def cloister_bell(self):
        print("(Ominous bell tolling)")

    def dematerialize(self):
        self._timestream.append(self._destination)
        print("(Nifty whirring sound)")

    def set_destination(self, dest):
        if dest in self._timestream:
            self.cloister_bell()
        self._destination = dest

    def engage(self):
        if self._destination in self._timestream:
            raise SpacetimeError("You should not cross your own timestream!")
        else:
            self.dematerialize()


tardis = Tardis()

# Should be fine
tardis.set_destination("7775/349x10,012/acorn")
tardis.engage()

# Also fine
tardis.set_destination("5136/161x298,58/delta")
tardis.engage()

# The TARDIS is not going to like this...
tardis.set_destination("7775/349x10,012/acorn")
tardis.engage()

Obviously, that last one is going to lead to our SpacetimeError exception being raised.

Let's look at that exception class declaration again.

class SpacetimeError(Exception):
    def __init__(self, message):
        super().__init__(message)

That's actually super easy to write. If you remember from our earlier exploration of classes, super().__init__() is calling the initializer on the base class, which is Exception in this case. We're taking the message passed to the SpacetimeError exception constuctor, and handing it off to that base class initializer.

In fact, if the only thing I'm doing is passing the message to the super(), class, I can make this even simpler:

class SpacetimeError(Exception):
    pass

Python handles the basics itself.

That's all we need to do, although as usual, there are many more tricks we can do with this. Custom exceptions are more than just a pretty name; we can use them to handle all sorts of unusual error scenarios, although that's obviously beyond the scope of this guide.

Review

Well, you've made it through our exploration of Python errors without being vaporized, deleted, upgraded, or accidentally dropped off in the wrong county, so three cheers for you! Let's review the essentials:

• Don't be afraid of exceptions in Python! We can use them to make our code much cleaner.
• Catch exceptions using a try...except block. (It's except, NOT catch!)
• Never use the diaper anti-pattern, which is a bare except: statement, or (often) an except Exception:
• The else block can be included after the last except, and only runs if there were no exceptions raised by the code in the try block.
• The finally block can be included at the end of the try...except statement, and is always run, even if we return in the midst of an except or else block.
• We "throw" an exception by raising it, via raise WhateverError("Our message")
• Inside an except block, we can bubble up (re-raise) an exception with a bare raise.
• We can create custom exception classes by deriving from the Exception class, or one of its many subclasses.

As always, the documentation reveals much, much more. I highly recommend checking it out:

• Python Tutorial: Errors and Exceptions
• Python Reference: Built-In Exceptions
• Python Reference: Built-In Exceptions - NotImplementedError
• Python Reference: Built-In Constants - NotImplemented
• PEP 3134: Exception Chaining and Embedded Tracebacks
• The Most Diabolical Python Antipattern

Thank you to deniska and grym (Freenode IRC #python) for suggested revisions.