Like the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press.

Python has a reputation for looking like magic, and that's probably due in part to the many forms a function can take: lambdas, decorators, closures, and more. A well-placed function call can do amazing things, without ever writing a single class!

You might say Functions Are Magic.

Functions Revisited

We already touched on functions in Data Typing and Immutability. If you haven't read that article yet, I'd recommend going back and taking a look now.

Let's look at a quick example of a function, just to make sure we're on the same page.

def cheer(volume=None):
    if volume is None:
        print("yay.")
    elif volume == "Louder!":
        print("yay!")
    elif volume == "LOUDER!":
        print("*deep breath*")
        print("yay!")


cheer()  # prints "yay."
cheer("Louder!")  # prints "yay!"
cheer("LOUDER!")  # prints "*deep breath* ...yay!"

Nothing surprising here. The function cheer() accepts a single parameter volume. If we don't pass an argument for volume, it will default to the value None instead.

What Is Functional Programming?

Those of us coming from object-oriented programming languages have learned to think of everything in terms of classes and objects. Data is organized into objects, alongside the functions responsible for accessing and modifying that data. Sometimes this works, but other times, classes just start feeling like too much boilerplate.

Functional programming is almost the opposite of this. We organize around functions, which we pass our data through. There are a few rules we have to follow:

• Functions should take only input, and produce only output.

• Functions should not have side effects; they should not modify anything external to themselves.

• Functions should (ideally) always produce the same output for the same inputs. There should be no state internal to the function that will break this pattern.

Now, before you go and rewrite all your Python code to be pure functional, STOP! Don't forget that one of the beautiful things about Python is that it's a multi-paradigm language. You don't have to choose one paradigm and stick to it; you can mix and match the best of each in your code.

In fact, we've already been doing this! Iterators and generators are both borrowed from functional programming, and they work quite well alongside objects. Feel free to incorporate lambdas, decorators, and closures too. It's all about choosing the best tool for the job.

In practice, we rarely can avoid side-effects all together. In applying the concept of functional programming to your Python code, you should focus more on being mindful and deliberate about side-effects, rather than just to avoid them altogether. Limit them to those situations where there is no better way to solve the problem. There's no hard-and-fast rule to rely on here; you will need to develop your own discernment.

Recursion

When a function calls itself, that's called recursion. This can be helpful when we need to repeat the entire logic of a function, but a loop is unsuitable (or just feels too cluttered).

NOTE: My example below is simplified to highlight recusion itself. This wouldn't actually be a situation where recursion would be the best approach; recusion is better when you need to repeatedly call complicated language on different pieces of data, such as when you're traversing a tree structure.

import random
random.seed()


class Villain:

    def __init__(self):
        self.defeated = False

    def confront(self):
        # Roll of the dice.
        if random.randint(0,10) == 10:
            self.defeated = True


def defeat(villain):
    villain.confront()
    if villain.defeated:
        print("Yay!")
        return True
    else:
        print("Keep trying...")
        return defeat(villain)


starlight = Villain()
victory = defeat(starlight)

if victory:
    print("YAY!")

The stuff related to random may look new. It's not really related to this topic, but in brief, we can generate random integers by first seeding the random number generator at the start of our program (random.seed()), and then calling random.randint(min, max), where min and max define the inclusive range of possible values.

The important part of the logic here is the defeat() function. As long as the villain is not defeated, the function calls itself, passing the villain variable. This will happen until one of the function calls returns a value. In this case, the value is returned up the recursive call stack, eventually getting stored in victory.

No matter how long it takes*, we'll eventually defeat that villain.

Beware Recursing Infinitely

Recursion can be a powerful tool, but it can also present a problem: what if we have no way to stop?

def mirror_pool(lookers):
    reflections = []
    for looker in lookers:
        reflections.append(looker)
    lookers.append(reflections)

    print(f"We have {len(lookers) - 1} duplicates.")

    return mirror_pool(lookers)


duplicates = mirror_pool(["Pinkie Pie"])

Clearly, this is going to run forever! Some languages don't provide a clean way to handle this — the function will just recurse infinitely until something crashes.

Python stops this madness a little more gracefully. As soon as it reaches a set recursion depth (usually 997-1000 times), it stops the entire program and raises an error:

RecursionError: maximum recursion depth exceeded while calling a Python object

Like all errors, we can catch this before things get out of hand:

try:
    duplicates = mirror_pool(["Pinkie Pie"])
except RecursionError:
    print("Time to watch paint dry.")

Thankfully, because of how I wrote this code, I don't actually need to do anything special to clean up the 997 duplicates. The recursive function never returned, so duplicates remains undefined in this case.

We might want to control recursion another way, however, so we don't have to use a try-except to prevent disaster. Within our recursive function, we can keep track of how many times it's been called by adding a calls parameter, and aborting as soon as it gets too big.

def mirror_pool(lookers, calls=0):
    calls += 1

    reflections = []
    for looker in lookers:
        reflections.append(looker)
    lookers.append(reflections)

    print(f"We have {len(lookers) - 1} duplicates.")

    if calls < 20:
        lookers = mirror_pool(lookers, calls)

    return lookers


duplicates = mirror_pool(["Pinkie Pie"])
print(f"Grand total: {len(duplicates)} Pinkie Pies!")

We still have to figure out how to get rid of 20 duplicates without losing the original, but at least the program didn't crash.

NOTE: You can override the maximum recursion level with sys.setrecursionlimit(n), where n is the maximum you want.

Nested Functions

From time to time, we may have a piece of logic which we want to reuse within a function, but we don't want to clutter up our code by making yet another function besides.

def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element, target):
        print(f"Using Element of {element} on {target}.")

    for element in elements:
        use(element, target)


use_elements("Nightmare Moon")

Of course, the trouble with an example this simple is that the usefulness is not immediately apparent. Nested functions become helpful when we have a large chunk of logic that we want to abstract down to a function for reuability, but we don't want to define outside of our principal function. If the use() function were considerably more complicated, and perhaps called from more than just the loop, this design would be justified.

Still, the simplicity of the example shows the underlying concept. It also brings up another difficulty. You will notice that we're passing target to the inner function, use(), each time we call it, and that feels rather pointless. Couldn't we just use the target variable that's already in local scope?

In fact, we could.

def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        print(f"Using Element of {element} on {target}.")

    for element in elements:
        use(element)


use_elements("Nightmare Moon")

Yet as soon as we try to modify that variable, we run into trouble:

def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        print(f"Using Element of {element} on {target}.")
        target = "Luna"

    for element in elements:
        use(element)

    print(target)


use_elements("Nightmare Moon")

Running that code raises an error:

UnboundLocalError: local variable 'target' referenced before assignment

Clearly, it is no longer seeing our local variable target. This is because assigning to a name, by default, shadows any existing names in enclosing scopes. So, the line target == "Luna" is trying to create a new variable limited to the scope of use(), and that hides (shadows) the variable target in the enclosing scope of use_elements(). Python sees this and assumes that, since we're defining target in the function use(), all references to that variable relate to that local name. That's not what we want!

The nonlocal keyword allows us to tell the inner function that we're working with the variable target from the enclosing local scope.

def use_elements(target):
    elements = ["Honesty", "Kindness", "Laughter",
                "Generosity", "Loyalty", "Magic"]

    def use(element):
        nonlocal target
        print(f"Using Element of {element} on {target}.")
        target = "Luna"

    for element in elements:
        use(element)

    print(target)


use_elements("Nightmare Moon")

Now, when all is said and done, we see the value Luna printed out. Our work here is done!

NOTE: If you're wanting to allow a function to be able to modify a variable that was defined the global scope (outside of all functions), use the global keyword instead of nonlocal.

Closures

Building on the idea of the nested function, and recalling that a function is treated no differently than any other object, we can create a function that actually builds and returns another function, called a closure.

def harvester(pony):
    total_trees = 0

    def applebucking(trees):
        nonlocal pony, total_trees
        total_trees += trees
        print(f"{pony} harvested from {total_trees} trees so far.")

    return applebucking


apple_jack = harvester("Apple Jack")
big_mac = harvester("Big Macintosh")
apple_bloom = harvester("Apple Bloom")

north_orchard = 120
west_orchard = 80  # watch out for fruit bats
east_orchard = 135
south_orchard = 95
near_house = 20

apple_jack(west_orchard)
big_mac(east_orchard)
apple_bloom(near_house)
big_mac(north_orchard)
apple_jack(south_orchard)

In this example, applebucking() would be the closure, because it closes over the nonlocal variables pony and total_trees. Even after the outer function terminates, the closure retains references to these variables.

The closure is returned from the harvester() function, and can be stored in a variable like any other object. It is the fact it "closes over" a nonlocal variable that makes it a closure per se; otherwise, it'd just be a function.

In this example, I'm using the closure to effectively create objects with state. In other words, each harvester remembers how many trees he or she has harvested from. This particular usage isn't strictly compliant with functional programming, but it's quite useful if you don't want to create an entire class just to store one function's state!

apple_jack, big_macintosh, and apple_bloom are now three different functions, each with their own separate state; they each have a different name, and remember how many trees they have harvested from. What happens in one closure's state has no effect on the others.

When we run the code, we see this state in action:

Apple Jack harvested from 80 trees so far.
Big Macintosh harvested from 135 trees so far.
Apple Bloom harvested from 20 trees so far.
Big Macintosh harvested from 255 trees so far.
Apple Jack harvested from 175 trees so far.

Easy as apple pie.

The Problem With Closures

Closures are essentially "implicit classes", because they put functionality and its persistent information (state) in the same object. There are, however, several unique disadvantages to closures:

• You cannot access the "member variables" as it were. In our example, I can never get to the total_trees variable on the apple_jack closure! I can only use that variable within the context of the closure's own code.

• The state of the closure is entirely opaque. Unless you know how the closure is written, you don't know what information it's keeping track of.

• Because of the previous two points, it is impossible to directly know when a closure has any state at all.

When using closures, you need to be prepared to handle these problems, and all the debugging difficulties they introduce. I recommend only using them when you need a single function to store a small amount of private state between calls, and only for such a limited period of time in the code that writing an entire class doesn't feel justified. (Also, don't forget about generators and coroutines, which may be better suited to many such scenarios.)

Basically, it's there. Closures can still be a useful part of your Python repitoire, so long as you use them with great care.

Lambdas

A lambda is an anonymous function (no name) made up of a single expression.

That definition alone is the reason many programmers can't imagine why they'd ever need one. What's the point of writing a function that lacks a name, basically making reuse completely impractical? Sure, you can assign a lambda to a variable, but at that point, shouldn't you have just written a function?

To understand this, let's take a look at an example without lambdas first:

class Element:

    def __init__(self, element, color, pony):
        self.element = element
        self.color = color
        self.pony = pony

    def __repr__(self):
        return f"Element of {self.element} ({self.color}) is attuned to {self.pony}"


elements = [
    Element("Honesty", "Orange", "Apple Jack"),
    Element("Kindness", "Pink", "Fluttershy"),
    Element("Laughter", "Blue", "Pinkie Pie"),
    Element("Generosity", "Violet", "Rarity"),
    Element("Loyalty", "Red", "Rainbow Dash"),
    Element("Magic", "Purple", "Twilight Sparkle")
]


def sort_by_color(element):
    return element.color


elements = sorted(elements, key=sort_by_color)
print(elements)

The main thing I want you to notice is the sort_by_color() function, which I had to write for the express purpose of sorting the Element objects in the list by their color. This is a bit of an annoyance, actually, since I won't be needing that function ever again.

Here's where lambdas come in. I can drop that entire function, and change the elements = sorted(...) line to:

elements = sorted(elements, key=lambda e: e.color)

Using a lambda allows me to delineate my logic exactly where I'm using it, and nowhere else. (The key= part is just indicating that I'm passing the lambda to the key parameter on sorted().)

A lambda has the structure lamba <parameters>: <return expression>. It can collect as many paramters as it likes, separated by commas, but it can only have one expression, the value of which is implicitly returned.

GOTCHA ALERT: Lambdas do not support type annotations (type hinting), unlike regular functions.

If I wanted to rewrite that lambda to sort by the name of the Element, instead of the color, I only need to change the expression part:

elements = sorted(elements, key=lambda e: e.name)

It's as simple as that.

Again, lambdas are chiefly useful whenever you need to pass a function with a single expression to another function. Here's another example, this time with more parameters on the lambda.

To set up this example, let's start with a class for a Flyer, which stores the name and the maximum speed, and returns a random speed for the flyer.

import random
random.seed()


class Flyer:

    def __init__(self, name, top_speed):
        self.name = name
        self.top_speed = top_speed

    def get_speed(self):
        return random.randint(self.top_speed//2, self.top_speed)

We want to be able to make any given Flyer object perform any flying trick, but putting all that logic into the class itself would be impractical...there are perhaps thousands of flying tricks and variants!

Lambdas are one way to define these tricks. We'll start by adding a function to this class that can accept a function as a parameter. We'll make the assumption that this function always takes a single argument: the speed at which the trick is performed.

    def perform(self, trick):
        performed = trick(self.get_speed())
        print(f"{self.name} perfomed a {performed}")

To use this, we create a Flyer object, and then pass functions to its perform() method.

rd = Flyer("Rainbow Dash", 780)
rd.perform(lambda s: f"barrel-roll at {s} mph.")
rd.perform(lambda s: f"flip at {s} mph.")

Because the lambda's logic is in the function call, it is a lot easier to see what's going on.

Recall that you are allowed to store lambdas in a variable. This actually can be helpful when you want the code to be this brief, but need some reuability. For example, assume we have another Flyer, and we want both of them to perform a barrel-roll.

spitfire = Flyer("Spitfire", 650)
barrelroll = lambda s: f"barrel-roll at {s} mph."

spitfire.perform(barrelroll)
rd.perform(barrelroll)

Sure, we could have written barrelroll as a proper single-line function, but by doing it this way, we saved ourselves a little bit of boilerplate. And, since we won't be using the logic again after this section of code, there's no point in having a full-blown function hanging out.

Once again, readability matters. Lambdas are excellent for short, clear fragments of logic, but if you have anything more complicated, you should definitely write a proper function.

Decorators

Imagine we want to modify the behavior of any function, without actually changing the function itself.

Let's start with a reasonably basic function:

def partial_transfiguration(target, combine_with):
    result = f"{target}-{combine_with}"
    print(f"Transfiguring {target} into {result}.")
    return result


target = "frog"
target = partial_transfiguration(target, "orange")
print(f"Target is now a {target}.")

Running that gives us:

Transfiguring frog into frog-orange.
Target is now a frog-orange.

Simple enough. But what if we wanted to add some extra fanfare to this? As you know, we really shouldn't put that logic in our partial_transfiguration function.

This is where decorators come in. A decorator "wraps" additional logic around a function, such that we don't actually modify the original function itself. This makes for code that is far more maintainable.

Let's start by creating a decorator for the fanfare. The syntax here might look a little overwhelming at first, but rest assured I'll break this down.

import functools


def party_cannon(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        print("Waaaaaaaait for it...")
        r = func(*args, **kwargs)
        print("YAAY! *Party cannon explosion*")
        return r

    return wrapper

You may have already recognized that wrapper() is actually a closure, which is created and returned from our party_cannon() function. We pass the function we're "wrapping", func.

However, we really don't know anything about the function we're wrapping! It may or may not have arguments. The closure's parameter list (*args, **kwargs) can accept literally any number of arguments, from zero to (practically) infinity. We pass these arguments in the same way to func() when we call it.

Of course, if there's some sort of mismatch between the parameter list on func() and the arguments passed to it through the decorator, the usual and expected error will be raised (which is obviously a good thing.)

Within wrapper(), we are calling our function whenever and however we want to with func(). I chose to do so between printing my two messages.

I don't want to throw away the value that func() returns, so I assign that returned value to r, and I make sure to return it at the end of my wrapper with return r.

Mind you, there are really no hard-and-fast rules for the manner in which you call the function in the wrapper, or even if or how many times you call it. You can also handle the arguments and return values in whatever manner you see fit. The point is to make sure the wrapper doesn't actually break the function it wraps in some unexpected way.

The odd little line just before the wrapper, @functools.wraps(func), is actually a decorator itself. Without it, the function being wrapped would essentially get confused as to its own identity, messing up our external access of such important function attributes as __doc__ (the docstring) and __name__. This special decorator ensures that doesn't happen; the function being wrapped retains its own identity, which is accessible from outside of the function in all the usual ways. (To use that special decorator, we had to import functools first.)

Now that we have our party_cannon decorator written, we can use it to add that fanfare we wanted to the partial_transfiguration() function. Doing so is as simple as this:

@party_cannon
def partial_transfiguration(target, combine_with):
    result = f"{target}-{combine_with}"
    print(f"Transfiguring {target} into {result}.")
    return result

That first line, @party_cannon is the only change we had to make! The partial_transfiguration function is now decorated.

NOTE: You can even stack multiple decorators on top of each other, one above the next. Just make sure each decorator comes immediately before the function or decorator it is wrapping.

Our usage from before hasn't changed at all:

target = "frog"
target = partial_transfiguration(target, "orange")
print(f"Target is now a {target}.")

Yet the output has indeed changed:

Waaaaaaaait for it...
Transfiguring frog into frog-orange.
YAAY! *Party cannon explosion*
Target is now a frog-orange.

Review

We've covered four aspects of functional "magic" in Python. Let's take a moment to recap.

• Recursion is when a function calls itself. Beware "infinite recursion"; Python won't let a recursion stack get more than approximate a thousand recursive calls deep.

• A nested function is a function defined in another function.

• A nested function can read the variables in its enclosing scope, but it cannot modify them unless you specify the variable as nonlocal first in the nested function.

• A closure is a nested function that closes over one or more nonlocal variables, and then is returned by the enclosing function.

• A lambda is an anonymous (unnamed) function made up of a single expression, the value of which is returned. Lambdas can be passed around and assigned to variables like any other object.

• Decorators "wrap around" another function to extend its behavior, without the function you're wrapping having to be directly modified.

You can read the documentation for more information about these topics. (You'll actually notice that nested functions and closures are rarely mentioned in the official documentation; they're design patterns, rather than formally defined language structures.)

• Python Reference: Compound statements — Function definitions

• Python Reference: Expressions — Lambda

• PEP 318: Decorators for Functions and Methods

• Python Standard Library: functools

Thanks to @deniska, @asdf, @SnoopDeJi (Freenode IRC), and @sandodargo (DEV) for suggested improvements.