What is this
The guide to set up a Python project for production deployment.
This guide assumes you are using Poetry for package management and adheres to the rules specified.
If you are not familiar with how to use Poetry, please read the following documentation and install it locally:
By following this guide, you can set up a Python project for production deployment using Poetry as the package manager, enforce code quality with linters and formatters, automate checks with GitHub Actions, configure your IDE to maintain consistency, and write comprehensive unit tests to ensure the project's reliability throughout the development process.
We recommend using Python version 3.9 or higher for your project.
The majority of this document was created using ChatGPT-4.
-
Sample Project
Steps
1. Project Setup/ Set up Local Environment with Poetry and Pyenv
- Create a project directory and navigate to it:
mkdir ${project_path}
cd ${project_path}
Use pyenv to manage local Python versions and ensure a consistent environment for all team members. To do this, include a .python-version file in the project.
Set the desired Python version for the project using pyenv:
pyenv install ${python_version}
pyenv local ${python_version}
Replace ${python_version} with the desired version, e.g., 3.9.7. This command will create a .python-version file in the project directory.
Ensure that all team members use the same Python version specified in the .python-version file.
Copy the provided pyproject.tool file to the project directory and install dependencies:
-
Copy pyproject.yml example and arrange by yourself
- For the local environment, use Poetry to create a .venv virtual environment in the project root. Place the following poetry.toml file in the project directory:
[virtualenvs]
path = ".venv"
in-project = true
- Execute the following commands to set up the virtual environment and install the required packages
poetry env use ${python_version}
poetry install
// Run command in the created environment.
poetry run <YOUR_COMMAND>
// Enter the created environment.
poetry shell
In the pyproject.toml file, we configure various modules for linting, formatting, and task running. Here's an explanation of these modules:
- Taskipy
Taskipy is a task runner that allows you to automate various tasks in your Python project. In this project, we use Taskipy to define and run tasks such as formatting, linting, and testing. The tasks are configured in the pyproject.toml file under [tool.taskipy.tasks].
For more information about Taskipy, check out the official documentation:
- Linters and Formatters
This project uses several linters and formatters to maintain code quality and consistency:
Black: A code formatter that automatically formats your code according to the PEP 8 style guide. The configuration is set in the pyproject.toml file under [tool.black].
Ruff: A linter that checks your code for style guide violations, potential bugs, and other issues. This covers so many rules inluding flake8’s one and isort’s one. The configuration is set in the pyproject.toml file under [tool.ruff].
Mypy: A static type checker that helps catch potential bugs and ensures that your code is type-safe. The configuration is set in the pyproject.toml file under [tool.mypy].
These linters and formatters are integrated into the project's tasks and the IDE settings, ensuring a consistent code style across the entire project.
2. Create Directory Structure
Create the necessary directories and files:
mkdir assets # you can write main codes in this directory
mkdir tests # you can write test codes in this directory
touch README.md
3. Write Sufficient Unit Tests with pytest
Ensure that you write comprehensive unit tests using pytest for all critical components of your project. Adequate test coverage helps maintain code quality and reduces the likelihood of introducing bugs during development or refactoring.
To write a unit test, create a unit test, create a new file in the tests directory with a name that follows the pattern test_*.py.
Within the test file, import the necessary modules and define test functions that follow the pattern def test_*():.
Use pytest's built-in assert statement to compare expected and actual values.
To run tests, execute the following command:
poetry run task test
Ensure that all tests pass before committing changes and creating a pull request.
4. Configure IDE Settings
Reflect the contents of the provided pyproject.toml file in the settings file related to your IDE.
For example, if you are using VSCode, add the following settings to .vscode/settings.json:
{
"python.formatting.provider": "black",
"python.formatting.blackArgs": ["--line-length", "102"],
"python.linting.flake8Enabled": true,
"python.linting.mypyEnabled": true,
"python.sortImports.args": ["--profile=black", "--line-length=102"],
"editor.codeActionsOnSave": {
"source.organizeImports": true,
"source.fixAll": true
}
}
With these settings, VSCode will automatically format and lint code using Black, Flake8, Mypy, and isort.
If you use IntelliJ/Pycharm, Please use File Watcher function for auto-running poetry run task fmt_lint
5. Set up CI(Auto lint/test) with GitHub Actions
Create the .github/workflows directory in the project root:
mkdir -p .github/workflows
Create a test_and_lint.yml file and add the following content:
name: python lint and test
on:
pull_request:
branches:
- main
paths:
- "YOUR_PROJECT/**.py"
- "tests/**.py"
- "poetry.toml"
- "poetry.lock"
- ".github/workflows/test_and_lint.yml"
jobs:
lint_and_test:
name: Lint and Test
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.11","3.10","3.9"]
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install Poetry
run: |
curl -sSL https://install.python-poetry.org | POETRY_HOME=$HOME/.poetry python3 -
echo "$HOME/.poetry/bin" >> $GITHUB_PATH
# cacheがうまく効かない環境があるのでその際は以下の設定はコメントアウトする
- uses: actions/cache@v2
id: venv_cache
with:
path: .venv
key: venv-${{ matrix.python-version }}-${{ hashFiles('**/poetry.lock') }}
- name: Install Dependencies
if: steps.venv_cache.outputs.cache-hit != 'true'
run: poetry install
- name: Python Test
run: poetry run task test
- name: Python Lint
run: poetry run task lint
-
sample
This configuration will automatically run poetry run task fmt_lint and poetry run task test on pushes and pull requests to the main branch.
6. Set up Stage/Prod Environment with Dockerfile
For the stage/prod environment, use a Dockerfile to build the execution environment. First, export the project's dependencies to a requirements.txt file using the following command:
poetry export --without-hashes -f requirements.txt -o requirements.txt
In the Dockerfile, use pip install to install the dependencies from the requirements.txt file. Ensure that the specified Python version in the Dockerfile is the same as the one used in the local environment.
- sample Dockerfile
By following these rules, you can ensure a consistent development environment for all team members and maintain a clean separation between local and stage/prod environments.
Additionally, it is recommended to integrate the configuration of this process into the GitHub Actions YAML for the next 7 Continuous Deployment (CD) steps, as it eliminates the need to work locally.
Top comments (0)