Python modules can be thought of as a Singleton design pattern. They are initialized once, the first time you import the module in the project from the interpreters' point of view. The next time you import the module the same instance is returned. This optimization makes sense, you do not want to re-compile the entire module every time it is imported. However, one common criticism against Singletons is that they are hard to test.
So, why are Singletons and, therefore, Python modules hard to test?
Readers be like:
What is this joker talking about, Python modules aren't hard to test?
Calm down, I will explain. As long as you are using modules just as a collection of functions and classes you will be fine. However, when you introduce any type of state in a module you will start to feel the pain as you write the tests. Ideally, we would have stateless modules but this is not always the case in the real world. Ever written a Flask server? I can almost guarantee you have some state baked into that module. You can make testing the module easier by for example pushing as much of the functionality into separate modules that can be tested. However, some state is more subtle, like a parameterized decorators.
The reason that I am writing this post now is that I had a fight with a caching decorator last week. It stored the state from previous tests which caused the next test to break. This also meant each test worked fine when running in isolation but failed when running the entire suit. Fun times!
The next step was to mock out the caching decorator. However, this can't be done in the module using the decorator as it is applied when importing the module. This gives me no time to swap it before it is applied. Instead, I have to mock it inside the module where I define the decorator. Again, this will cause some problems, what if the module using the decorator already has been imported? It is a Singleton and will, therefore, be using the real decorator. The solution, importlib.reload which is a part of the Python standard library. This method allows you to force a reload of a module, which will have the same effect as if it was imported for the first time.
Let us look at a short example of how reload works. First, we create a file, my_module.py:
# my_module.py
x = 1
Secondly, we have a file example.py:
# example.py
from importlib import reload
import my_module
print(f'Original: {my_module.x}')
my_module.x = 12
print(f'Changed: {my_module.x}')
reload(my_module)
print(f'Reloded: {my_module.x}')
Running example.py gives the output:
Original: 1
Changed: 12
Reloded: 1
This is a trivial example but it shows how reload resets the state of the module.
Now back to the caching decorator. This time we will use the import behavior to our advantage. In Python, we can to monkey patching; swapping out classes, methods, and functions at run-time by overriding their name. First, we monkey patch the caching decorator to do no caching. Secondly, we use ìmportlib.reload to reload the module using the decorator. When the module is reloaded it imports are patched version of the caching decorator. Now we can run the entire suite with caching turned off.
As I am using pytest, I ran this behavior as a part of setup_module. I also added reload the decorator module and the one using it as a part of teardown_module, resetting everything to their ordinary behavior. By doing this I contain the mocked caching to a single test module and can freely test the caching itself in another test module. YAAAY!
Top comments (0)