Completely Type-Safe Dependency Injection in Python

Simply put, dependency injection is a collection of programming techniques that enables software components to have their dependencies replaced, thereby increasing re-usability, for example by allowing a database module to depend on a connection string in order that it can connect to multiple databases. Dependency injection also improves testability because complicated dependencies such as database connections can be easily replaced with mocks.

In this post we'll study dependency injection in Python. We'll see how it can be made completely type safe using functional programming, specifically with a modern functional programming library called pfun.

Dependency injection in Python is typically implemented using monkey patching. In fact, monkey patching as a form of providing dependencies is so common that the unittest.mock.patch function was added to the standard library in Python 3.3.

While monkey patching is a simple technique, it often leads to rather complex patching scenarios, where it's tricky to figure out what and how to patch. Moreover, it can be tricky to achieve complete type safety with monkey patching on both the dependency consumer and provider side. Finally, there are no straight-forward ways of making sure that the dependency provider provides all required dependencies, often leading to statements such as if dependency is not None.

An alternative (and potentially even simpler) method for dependency injection is the following: function arguments. In fact, this approach is so simple that using the term "dependency injection" to describe it seems almost pretentious. An attractive feature of implementing dependency injection with functions that take arguments is that it is completely type safe by default (provided that you use type annotations of course). Moreover, you can tell a function's dependencies simply by reading its signature, making it totally clear what needs to be provided, and in fact making it impossible to not provide all required dependencies.

The main drawback from using function arguments as your "injection" mechanism, is that functions that call functions that have "injected dependencies" now need to take those dependencies as arguments themselves.

For example, consider a function connect that returns a connection to a database given a connection_str as argument:

def connect(connection_str: str) -> Connection:
    ...

To achieve the promise of "dependency injection", every function that calls connect must now take a connection_str parameter in addition to its other parameters, in order that the calling function itself can be used against different databases:

def get_user(connection_str: str, user_id: int) -> User:
    connection = connect(connection_str)
    return connection.get_user(user_id)

For functions with many dependencies, this quickly becomes very tedious.

Thankfully, we can use functional programming to improve the situation by a margin: Notice how the only use get_user has of connection_str is to pass it to connect. With functional programming we can abstract this pattern of functions taking arguments only to pass them to other functions into some general 'plumbing' functions. This will alow us to write get_user without mentioning the connection_str at all!

To keep things nice and readable (and save us some typing), lets define a type-alias that represents functions that require dependencies:

from typing import TypeVar, Callable


R = TypeVar('R')
A = TypeVar('A')
Depends = Callable[[R], A]

In words, Depends[R, A] is a function that depends on something of type R to produce something of type A. For example, our previous connect function is a Depends[str, Connection] value: it's a function that depends on a str to produce a Connection.

Simple enough, but how do we use the Depends represented by connect in get_user without explicitly taking connection_str as a parameter and calling connect? To do that, let's introduce our first plumbing function map_:

B = TypeVar('B')


def map_(d: Depends[R, A], f: Callable[[A], B]) -> Depends[R, B]:
    def depends(r: R) -> B:
        return f(d(r))
    return depends

map_ precisely implements the "passing of parameters" plumbing we were talking about earlier: it creates a new Depends value that calls d, and passes the result to a function f. This allows us to write get_user as follows:

def get_user(user_id: int) -> Depends[str, User]:
    return map_(connect, lambda con: con.get_user(user_id))

And voila: we have type-safe dependency injection using only Depends (which is just a type-alias for functions of 1 argument) and map_ (which is just a function that composes Depends values with functions). Realise that we could keep applying map_ to Depends values to return a final Depends all the way back to where we make desicions about what concrete connection strings to pass in (probably in the main section of our program). The pattern we've discovered here seems simple enough that it would be surprising if we were the firsts to think of it, and sure enough it's in fact a common pattern in functional programming called the reader effect.

That's all well and good, but what happens when we want to map_ functions that return new Depends values? For example, let's imagine that in addition to reading user data from databases, our application also needs to call an HTTP api with the user data. Doing so requires authentication credentials that we want to inject. The function that calls the api might look like this:

from typing import Tuple


Auth = Tuple[str, str]


def call_api(user: User) -> Depends[Auth, bytes]:
    def depends(auth: Auth) -> bytes:
        ...
    return depends

We might try to use map_ to pass the result of the Depends value returned by get_user to call_api. But in doing so we would end up with a Depends[str, Depends[Auth, bytes]]:

if __name__ == '__main__':
    user_1: Depends[str, User] = get_user(user_id=1)
    call_api_w_user_1: Depends[str, Depends[Auth, bytes]] = map_(user_1, call_api)

When we want to call call_api_w_user_1, we need to first supply the connection string, and then the auth information:

result: bytes = call_api_w_user_1('user@prod')(('prod_user', 'pa$$word'))

This might not seem like big issue in this example, but notice that we might apply map_ over several functions that produces Depends values with the same dependency type, resulting in a situation where we have to pass in the same dependency many times. We might also have a varying number of dependencies if we use map_ in a loop, leading to a situation where we don't know how many times we need to call the final Depends!

To fix this situation, let's introduce another plumbing function that can pass dependencies to both a Depends value, and a Depends value returned by a function that's being mapped. We'll call it and_then since it chains together results of Depends values with functions that return new Depends values:

from typing import Any


R1 = TypeVar('R1')


def and_then(d: Depends[R, A], f: Callable[[A], Depends[R1, B]]) -> Depends[Any, B]:
    def depends(r: Any) -> B:
        return map_(d, f)(r)
    return depends

The observant reader will notice that there's a big problem with the typing of our and_then function: it returns a Depends[Any, B]! Here you might object: 'But wait a minute, I though you said this was "completely type safe"'! The reason for this epic fail is that the r parameter passed to the depends function must be passed to both d, which means it must be of type R, and the Depends returned by f, which means it must be of type R1. In other words, the dependency must be of both type R and R1 at the same time. In our example, using and_then to combine get_user and call_api should result in a type that is both a str and a Tuple[str, str]:

call_api_w_user_1: Depends[?, bytes] = and_then(get_user(user_id=1), call_api)

Such a type is called an intersection type and unfortunately it's not supported by the Python typing module (yet). So without introducing third party libraries, this is the best we can do.

This is where the library pfun comes into the picture. In pfun, map_ and and_then are instance methods of a Depends type, but the idea is exactly the same:

from pfun import Depends


def connect() -> Depends[str, Connection]:
    ...

def get_user(user_id: int) -> Depends[str, User]:
    connect().map(lambda con: con.get_user(user_id))

def call_api(user: User) -> Depends[Auth, bytes]:
    ...

call_api_w_user_1 = get_user(user_id=1).and_then(call_api)

pfun provides mechanisms for combining Depends values with different dependency types when using MyPy with one small requirement: the dependency type must be a typing.Protocol. The reason is that intersections of protocol types are simply a new protocol that inherits from both:

from typing import Protocol


class P1(Protocol):
    pass


class P2(Protocol):
    pass


class Intersection(Protocol, P1, P2):
    pass

This means that when using and_then with Depends values that have protocol types as dependencies, pfun can infer an intersection type automatically. Let's rewrite our example to take advantage of this feature:

from typing import Protocol, Tuple


class HasConnectionStr(Protocol):
    connection_str: str


class HasAuth(Protocol):
    auth: Tuple[str, str]


def connect() -> Depends[HasConnectionStr, Connection]:
    ...


def get_user(user_id: int) -> Depends[HasConnectionStr, Connection]:
    connect().map(lambda con: con.get_user(1))


def call_api(user: User) -> Depends[HasAuth, bytes]:
    ...


call_api_w_user_1 = get_user(user_id=1).and_then(call_api)

The type of call_api_w_user_1 in this example will be Depends[pfun.Intersection[HasConnectionStr, HasAuth], bytes]. The dependency type pfun.Intersection ensures that call_api_w_user_1 must be
called with an argument that fulfills both the HasConnectionStr protocol and the HasAuth protocol. In other words, this would be a type error:

class Dependency:
    def __init__(self):
        connection_str = 'user@prod'


call_api_w_user_1(Dependency())  # MyPy Error!

This compositional nature of and_then means that complex dependency types are built from simple dependency types automatically, which means you can always figure out which dependencies a part of your application has, simply by inspecting it's return type. To learn more about functional programming with pfun check out some of the other posts in the series, the documentation or the github repo.

Background vector created by macrovector - www.freepik.com