Recently I had to add python3.8
for our Python linter (the strictest one in existence): wemake-python-styleguide
. And during this straight-forward (at first look) task, I have found several problems with test coverage that were not solved in Python community at all.
Let's dig into it.
Environment-based logic
The thing about this update was that python3.8
introduced a new API.
Previously we had visit_Num
, visit_Str
, visit_NameConstant
methods (we don't really care about what they do, only names are important), but now we have to use visit_Constant
method instead.
Well, we will have to write some compatibility layer in our app to support both new python3.8
, previous python3.6
, and python3.7
releases.
My first attempt was to create a route_visit
function to do the correct thing for all releases. To make it work I also had to define PY38
constant like so:
import sys
PY38 = sys.version_info >= (3, 8)
if PY38:
route_visit = ... # new logic: 3.8+
else:
route_visit = ... # old logic: 3.6, 3.7
And it worked pretty well. I was able to run my tests successfully. The only thing broken was coverage. We use pytest
and pytest-cov
to measure coverage in our apps, we also enforce --cov-branch
and --cov-fail-under=100
policies. Which enforce us to cover all our code and all branches inside it.
Here's the problem with my solution: I was either covering if PY38:
branch on python3.8
build or else:
branch on other releases. I was never covering 100% of my program. Because it is literally impossible.
Common pitfalls
Open-source libraries usually face this problem. They are required to work with different python versions, 3rd party API changes, backward compatibility, etc. Here are some examples that you probably have already seen somewhere:
try:
import django
HAS_DJANGO = True
except ImportError:
HAS_DJANGO = False
Or this was a popular hack during 2
/ 3
days:
try:
range_ = xrange # python2
except NameError:
range_ = range # python3
With all these examples in mind, one can be sure that 100% of coverage is not possible.
The common scenario to still achieve a feeling of 100% coverage for these cases was:
- Using
# pragma: no cover
magic comment to exclude a single line or a whole block from coverage - Or writing every compatibility related check in a special
compat.py
that were later ommited from coverage
Here's how the first way looks like:
try: # pragma: no cover
import django
HAS_DJANGO = True
except ImportError: # pragma: no cover
HAS_DJANGO = False
Let's be honest: these solutions are dirty hacks. But, they do work. And I personally used both of them countless times in my life.
Here's the interesting thing: aren't we supposed to test these complex integrational parts with the most precision and observability? Because that's where our application breaks the most: integration parts. And currently, we are just ignoring them from coverage and pretending that this problem does not exist.
And for this reason, this time I felt like I am not going to simply exclude my compatibility logic. I got an idea for a new project.
Conditional coverage
My idea was that # pragma
comments can have more information inside them. Not just no cover
, but no cover when?
. That's how coverage-conditional-plugin
was born. Let's use it and see how it works!
First, we would need to install it:
pip install coverage-conditional-plugin # pip works, but I prefer poetry
And then we would have to configure coverage
and the plugin itself:
[coverage:run]
# Here we specify plugins for coverage to be used:
plugins =
coverage_conditional_plugin
[coverage:coverage_conditional_plugin]
# Here we specify our pragma rules:
rules = # we are going to define them later.
Notice this rules
key. It is the most important thing here. The rule (in this context) is some predicate that tells: should we include lines behind this specific pragma
in our coverage or not. Here are some examples:
[coverage:coverage_conditional_plugin]
# Here we specify our pragma rules:
rules =
"sys_version_info >= (3, 8)": py-gte-38
"sys_version_info < (3, 8)": py-lt-38
"is_installed('django')": has-django
"not is_installed('django')": has-no-django
It is pretty clear what we are doing here: we are defining pairs of predicates to include this code if some condition is true and another code in the opposite case.
Here's how our previous examples would look like with these magic comments:
import sys
PY38 = sys.version_info >= (3, 8)
if PY38: # pragma: py-lt-38
route_visit = ... # new logic: 3.8+
else: # pragma: py-gte-38
route_visit = ... # old logic: 3.6, 3.7
What does it say? If we are running on py-lt-38
ignore if PY38:
part. But, cover else:
case. Because it is going to be executed and we know it. And we need to know how good did we cover it. On the other hand, if we are running on py-gte-38
then cover if PY38:
case and leave else:
alone.
And we can test that everything works correctly. Let's add some nonsense into our PY38
branch to see if it is going to be covered by python3.8
build:
As we can see: green signs show which lines were fully covered, the yellow line indicates that branch coverage was not full, and the red line indicates that the line was not covered at all. And here's an example of grey or ignored lines under the opposed condition:
Here you can find the full real-life source code for this sample.
And here's one more example with django
to show you how external packages can be handled:
try: # pragma: has-no-django
import django
HAS_DJANGO = True
except ImportError: # pragma: has-django
HAS_DJANGO = False
We use the same logic here. Do we have django
installed during tests (we have a little helper function is_installed
to tell us)? If so, cover try:
. If not, cover except ImportError:
branch. But always cover something.
Conclusion
I hope you got the idea. Conditional coverage allows you to add or ignore lines based on predicates and collecting required bits of coverage from every run, not just ignoring complex conditions and keeping our eyes wide shut. Remember, that the code we need to cover the most!
By the way, we have all kinds of helpers to query your environment:
-
os_environ
for env variables -
patform_release
andplatform_version
for OS-based values -
pkg_version
that returns package version information (as its name says) - and many others!
This little plugin is going to be really helpful for library authors that have to deal with compatibility and unfixed environments. And coverage-conditional-plugin
will surely cover their backs! Give it a star on Github if you like this idea. And read the project docs to learn more.
Top comments (0)