Tools to automate Python Tests

Heya fellows,

The code of this post can be found on Github (see here).

When projects grow a good test suite gives you confidence that new code you add don't cause parts of the application to break. It improves the maintainability of the project. The complexity of small projects is low and tests are only necessary when the size of the project increases. But for the sake of this tutorial we will write a small test to demonstrate the usage of pytest, tox and github actions.

We will store the tests within a separate folder. Here's the current structure of the project.

├── LICENSE
├── README.md
├── setup.py
├── tihttp.py
├── tests
│  ├── __init__.py
│  └── test_tihttp.py

We place a test_tihttp.py file within the tests folder. It will contain the test suite. The main() function which contains the logic for triggering the flags has to be imported from the tihttp.py file.

from tihttp import main

def test_GET_body(capsys):
    main(["-B", "http://jsonplaceholder.typicode.com/todos?userId=1"])
    captured = capsys.readouterr()
    result = captured.out
    with open("tests/jsonplaceholder.json", "r") as f:
        output = f.read()
    assert result == output

We compare the response of a GET request to an API with the expected JSON data. We place a jsonplaceholder.json file into the same folder which contains our expected output. Then we install pytest and let it execute the test.

$ pip install pytest
$ pytest

Our test passes. To increase the tests verbosity -v is a useful flag. Furthermore, I like to use the -s flag to see the captured output. Next we add pytest to our extra requirements in the setup.py file.

extras_require={
    'dev': [
        'pytest', # pip install tihttp[dev]
    ],
},

This gives us the possibility to install extra dependencies (testing, linting tools etc.) easily by adding a [dev] to the package name.

$ pip install .[dev]            # local install
$ pip install tihttp[dev]       # remote install, PyPI repo

We tested all of this with python 3.7.3. But how does our application behave when executed on a different interpreter version? So let's test it against different Python versions! We use tox. It let's us run tests in multiple virtual envs.

$ pip install tox

Tox needs a recipe to know which virtualenv/commands to create/execute. This recipe is named tox.ini.

[tox]
envlist = py36,py37,py38,py39

[testenv]
deps =
    pytest
commands =
    pytest

If some of the Python interpreters are missing on your system, install them from the deadsnakes archive:

$ sudo add-apt-repository ppa:deadsnakes/ppa
$ sudo apt install python3.5 python3.6 python3.7 python3.8 python3.9

Now let's test across different interpreters!

$ tox

If you wanna test against a specific environment or execute only one file, then type:

$ tox -e py38
$ tox -e py38 -- test/main_test.py   # executes only a single test

Ok, we did the test locally, but when working in a team using continuous integration is pretty convenient. We set a integrate.yaml file up within a .github/workflows directory to tell github actions what jobs to execute. The following github actions file will test across different platforms and Python versions.

name: Python package

on: [push]

jobs:
  run:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        python-version: [3.6, 3.7, 3.8, 3.9]
        os: [ubuntu-latest, macos-latest, windows-latest]
    steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}
      - name: Cache pip
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
            ${{ runner.os }}-
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install flake8 pytest pytest-cov
          pip install -r requirements.txt
      - name: Lint with flake8
        run: |
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
      - name: Test with pytest
        run: |
          pytest

Don't be intimidated by the length of this job. It's just illustrating how powerful Github workflows can be. 🥰