Automate the boring stuff with Continuous Integration (CI), Continuous Deployment (CD)
I first learned about Travis-CI through venturing out into GitHub to checkout several open source projects. I've seen how they have continuous integration workflows set up test their code on every commit, branch, and pull request.
That seemed so convenient so I went off to try it myself.
I've documented what I've learned below to help get you started!
Choosing a continuous integration service
What to consider when creating a Python3 CI/CD pipeline
- Python Community on GitHub - support for all python3 π versions.
- Choose your testing framework - Use it with any popular testing framework: pytest, unittest, etc.
- Open vs. closed source - I recommend whenever possible to choose an open source option. The below providers generally offer zero cost options for new open source projects with reasonable build π constraints. If you go closed source, youβre going to needs to fork over some π΅.
Common providers
- Travis-CI - my first and has served me well
- Circle CI - seems to have a little saner syntax and finer configuration relative to Travis-CI
- Gitlab - leans open source where the others are closed source
- GitHub Actions - newer, native to GitHub, and offers Linux, Windows, and MacOS builds.
My learning started with Travis-CI and am using it since I am familiar with it. I haven't done a detailed comparison of the tools and it fits the needs of simpler python CI/CD projects. I want to look into GitHub actions, now that itβs coming out of Beta.
You are free to use what you prefer, feel is right, or would like to start with. This article is not a comparison, but rather an introduction into what the process looks like with one CI service.
Start small and build up what you learn. You will make progress, run into challenges, figure some out. Remember to sleep!!!!
Photo credit: 4geeksacademy.co/feelings-learning-coding/
It's an exhilirating, rewarding, and sometimes stressful day to day. π¬
There isn't a wrong option on your path to learning more about these tools and process. Where you start doesn't have to be where you are in the future. My recommendation is that you pick one, don't over think it, and get it up and working!
Put your work out there and connect with others. You don't have to create something from scratch. You can help your favorite existing projects set up or improve their CI/CD pipeline.
It's better when we build together. πππππ
Knowledge compounds and you'll marvel at where you're at soon. πΊοΈ π π»
Prerequisites
You should have basic knowledge about Github, Python, testing frameworks, and how to use the Terminal. Knowledge of Pipenv (Pipfiles) and pytest is a plus, but not required. CI/CD tools are a great way to learn both. We will set up a Travis-CI account as well.
YAML Basics
Most CI services interact with your repositories using a YAML configuration file.
In case you need a tune up on YAML, here is a recommended link:
Note: Ansible uses the YAML format, and is is another useful tool. A more detailed guide is an encouraged topic for another time.
Travis-CI YAML Example
Here is a full example .travis.yml
file from one of my projects. We will break down each section.
# .travis.yml
dist: xenial
language: python
cache: pip
python:
- "3.6"
- "3.7"
- "3.8"
- "nightly"
matrix:
allow_failures:
- python: "nightly"
install:
- pip install pipenv --upgrade-strategy=only-if-needed
- pipenv install --dev
script:
- bash scripts/test.sh
after_script:
- bash <(curl -s https://codecov.io/bash)
Let's explain each section
dist: xenial
dist is where the Ubuntu Release codename is specified. xenial
is Ubuntu 16.04 LTS codename - see: http://releases.ubuntu.com/ for a full list.
This specifies the base operating system used for the rest of workflow.
language: python
language is the programming language used. This tutorial uses python, but it very well could be cpp
, go
, rust
, etc.
cache: pip
cache allows for python package versions to be stored between runs, to speed up sequential builds. Cache can apply to more than just python packages.
python:
- "3.6"
- "3.7"
- "3.8"
- "nightly"
python, given the above language specification, is a key for a sequence of python versions to perform builds against. Generally most CI tools use the latest bug release version for each minor version. The build logs will tell you the specific versions.
matrix:
allow_failures:
- python: "nightly"
matrix allows for modifications in the above build sequence. In this case, the allow failures key is specifies a reference to the python sequence above, and has the value of "nightly"
, meaning that that version is allowed to fail. Think of it as early detection and future python development. *Note: there are also development versions of each python version, such as "3.8-dev"
. These are used when python version are in release candidate or beta stages.
install:
- pip install pipenv --upgrade-strategy=only-if-needed
- pipenv install --dev
install is the first section that is performed inside each of the build targets (virtual environments for each of the python versions above).
Each item in the sequence is executed in order!
Pipenv and Pipfiles
In this example, we are using pipenv
and a Pipfile
.
Most Pipfiles have production dependencies and development dependencies. You will often find the packages required to run test cases or build documentation under the dev-packages
section. Below is an example:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
mypy = "*"
black = "*"
jupyter = "*"
better-exceptions = "*"
pytest = "*"
pytest-cov = "*"
isort = "*"
flit = "*"
autoflake = "*"
mkdocs = "*"
mkdocs-material = "*"
[packages]
pydantic = "==0.32.2"
The command pip install --dev
installs all packages under [packages]
and [dev-packages]
. pip install
would only install the packages under [packages]
.
Requirement.txt files
Note: It is also common to use
pip
with arequirements.txt
file.
pip install -r requirements.txt
Python.org now recommends pipenv
as part of its general tool recommendations.
Optional aside about different virtual environment options
Pipenv
has the advantage of providing a mechanism to install your packages into a virtual environment without another package. There are many alternative - one simple alternaive is venv
.
A newer alternative is poetry
(https://poetry.eustace.io/). poetry
aims to facilitate what both requirements.txt and Pipfiles do not bundle, and that is released dependencies when deploying to PyPi. You will typically see flit
and a pyproject.toml
file, or a setup.py
file with redundant depedency information. That redundancy is what poetry
is aiming to improve upon.
Now lets help you get your CI/CD setup
1. Get setup to install my python package project template
Ensure you have a compatible python π environment on your computer.
python3 --version
python3 -m pip --version
python3 -m pytest --version
If you are missing any of the above, you can add the packages to your main python install.
Recommended reading on Python main install vs. altinstall.
If you only have or want one python3 version on your machine, there is no harm to run the following commands. There are nuances between Ubuntu versions (16.04 vs. 18.04 vs. latest) on which python3 version is the default install.
sudo apt-get update
sudo apt-get install python3-pip
sudo apt-get install python3-pytest
Now that we have confirmed your base python setup, letβs go ahead and ensure the pipenv and cookiecutter python packages are installed.
python3 -m pip install --user pipenv
python3 -m pip install --user cookiecutter
2. Create a Python project and setup the Pipenv virtual environment
# make sure your path finds --user installs
## add `export PATH="$HOME/.local/bin:$PATH"`
## to your ~/.bashrc, ~/.zshrc file on linux
cookiecutter https://github.com/iancleary/pypackage
# For the sake of brevity, repos on GitHub can just use the 'gh' prefix
cookiecutter gh:iancleary/pypackage
`
Fill out the wizard!
Use cd new-directory
to change into the new directory you just created (replace new-directory
)
Run pipenv
to display it's options. Take a look at whatβs offered.
Next, run pipenv install --dev
to install the production and development dependecies specified in the Pipfile
.
Run pipenv shell
to load the virtual environment.
3. Run the tests locally and make sure they pass
I recommend you open up VS code or your text editor of choice to view the folder structure
Execute ./scripts/tests.sh
from within the directory's Pipenv.
This command executes a bash script that does several things:
- runs
pytest
to check test cases and check test coverage - checks formatting with the
black
package - runs static type checking against your code base with
mypy
- checks import sorting from standard lib, your application, and custom packages
These are all things that should be relegated to scripts to allow you to free your mind from simple formatting and other draining tasks. Let it do that work for you!
4. Push the directory to a remote git repo
I recommend you follow the guide on the Github Help page:
5. Connect Travis-CI to the repo
Again, the documentation of both GitHub and Travis-CI are great to follow for this step.
Head over to Travis-CI and sign in with your GitHub account:
Once you login to Travis-CI and Enable GitHub Access, you'll want to configure Travis from your Github Settings page.
6. Now we are configured, let's start a build
Make a simple change to the repo, either a commit, or open a new branch to start a build!
Grab your πΏ and let's watch the CI process start and complete! π
Itβs very cool to watch the process.
The key to remember is the repeatability of the test suite.
Once complete, you will have your new project added to the Travis-CI dashboard. The three red boxes below are:
- The connected repo that is currently selected.
- The action that started a build (commit or pull request)
- The build matrix visualized, with links to view more details.
7. Happy coding with CI/CD pipelines ππ
Your setup should give you and others confidence that the code will work under those conditions. If itβs an open source repo, there is no taking your word for it. The logs are right there!
You will not want to go back to your life before it!
It is such a quality improvement!
Where to go from here?
- Start offloading and automating your testing, deployments, and other activities. There is so many opportunites to tie in different hooks or features, this is just the beginning.
Some examples of what else can you automate
Pypi publishing
I haven't done much in this space yet, but I've seen other projects successfully use the flit python package.
If that is your next step, checkout flit
. I know there are some other good tutorials out there to get started inside and outside of a CI/CD pipeline.
Jupyter Notebook image deployment with CI/CD to Docker Hub
I've been using Jupyter notebooks and Docker more and more recently. I significantly improved the maintenance model of my personal image by setting up an automated testing and deployment pipeline using Travis CI. Check out my personal Jupyter notebook repository!
Checkout the Docker image is iancleary/personal-notebook.
bash
docker pull iancleary/personal-notebook
Amazon EC2 instance deployments
I also am in the process of learning how to leverage continuous deployment onto Amazon EC2 instances.
You can find my notes on connecting AWS CodeDeploy and Travis CI using a Flask project.
This example is a proof of concept, and I'm excited to use this workflow more in the future.
References
- I learned a lot from https://github.com/tiangolo/fastapi and credit them as authors within my cookicutter pypackage AUTHORS.md. This is not only nice to do, but is required by many open source licenses.
Top comments (0)