DEV Community

Cover image for Creating a Dockerfile to Perform pre-commit
sc0v0ne
sc0v0ne

Posted on • Edited on

Creating a Dockerfile to Perform pre-commit

#docker #python #bash #linter

One of the most used tools within the development flow is the linter. With the linter we can organize and standardize so that developers of the same project always maintain the same standard.

Using as an example, you are on a project with different developers. There is a need to standardize the same code for everyone. From there you choose a linter and all the devs start using it. However, within the project there will not be just one type of file from a specific language, there will be several. You will need to install another linter for another language again. A big effort, right?

Imagine that there is a tool where you can unify all these linters. Could you imagine?
If you imagined pre-commit, congratulations, you are up to date with the technologies, but if you don't know it, don't worry. I will explain below.

Pre-commit

With pre-commit we can unify these tools as given in the example above, using Git hooks, we can be one step ahead of the commit by detecting any inconvenience in the code within development. Having this tool in hand, we define a file called .pre-commit-config.yaml. With it we add hooks for the tools that will pre-evaluate the code. But below it will be explored in more detail.

Below is the project. Don't forget to leave a star on this incredible project.

GitHub logo pre-commit / pre-commit

A framework for managing and maintaining multi-language pre-commit hooks.

build status pre-commit.ci status

pre-commit

A framework for managing and maintaining multi-language pre-commit hooks.

For more information see: https://pre-commit.com/


View on GitHub


Tutorial

Starting the tutorial, I will be using 3 linters:

If you want to follow the explanation along with the project, you can find it at the link below:

GitHub logo sc0v0ne / learning_precommit

Learning precommit

Start Application
uvicorn  src.app.main:app --reload
Enter fullscreen mode Exit fullscreen mode
Run Tests
./src/tests/test_main.py
Enter fullscreen mode Exit fullscreen mode
Run Pre-commit
./src/scripts/linter.sh
Enter fullscreen mode Exit fullscreen mode

View on GitHub


Creating a simple project

Guys, remembering this tutorial and the pre-commit, the codes used are simple with a focus on teaching.

Let's create an empty folder, then open it in the IDE (Integrated development environment). From the directory where you start the IDE, you will create a new folder called src. Inside it you will create two more, one being app and the other tests. Inside each one you will create a file called init.py.

Inside app we will create a new file called main.py. This file will be our initial start of the application, we will use FastAPI to help. Inside the file you will add the following code:

src/app/main.py 
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"message": "Hey, Python!!!!!"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
    return {"item_id": item_id, "q": q}
Enter fullscreen mode Exit fullscreen mode

In this code we have two paths to the API.

Now pay attention to the following code, let's create a new directory. This directory will be to simulate Python's Alembic. When we have a service using a database that needs a migration manager. This directory was created without any external connection or use. It is being created to apply the use of linter, simulating a real situation. Inside the app, we will create a directory called alembic, it has the following file structure.

src/
....app/
........__init__.py
........main.py
........alembic/
...............versions/
.......................1975ea83b712_create_account_table.py
................env.py
................README
................script.py.mako
Enter fullscreen mode Exit fullscreen mode
src/app/alembic/versions

1975ea83b712_create_account_table.py

"""create account table
Revision ID: 1975ea83b712

Revises:

Create Date: 2011-11-08 11:40:27.089406
"""

# Example File

# revision identifiers, used by Alembic.

revision = '1975ea83b712'

down_revision = None

branch_labels = None

from alembic import op
import sqlalchemy as sa

def upgrade():
    pass

def downgrade():
    pass
Enter fullscreen mode Exit fullscreen mode
src/app/alembic/
  • env.py
  • README
  • script.py.mako

Now let's create a new file inside the tests directory, called test_main.py. This way we can check if our paths are passing the tests. Inside the test_main.py file, you will add the following code.

src/tests/test_main.py 
from fastapi.testclient import TestClient
from app.main import app

client = TestClient(app)

def test_read_root():
    response = client.get("/")
    assert response.status_code == 200
    assert response.json() == {"message": "Hey, Python!!!!!"}

def test_read_item():
    response = client.get("/items/42?q=test")
    assert response.status_code == 200
    assert response.json() == {"item_id": 42, "q": "test"}
Enter fullscreen mode Exit fullscreen mode

Outside the src directory, you will create a file called requirements.txt. Inside it you will add the following packages.

requirements.txt 
fastapi==0.95.2

uvicorn==0.22.0

pytest==7.4.0

httpx==0.24.1
Enter fullscreen mode Exit fullscreen mode

Time of truth, is it working?

First let's run our application.

 uvicorn  src.app.main:app --reload

# Output
INFO:     Will watch for changes in these directories: ['/home/example/Workspace/learning_precommit']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [50974] using StatReload
INFO:     Started server process [50976]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
Enter fullscreen mode Exit fullscreen mode

Let's access our URL.

Image create by author

Alright, let's go through the tests in our application.

pytest ./src/tests/test_main.py

#output
============================================ test session starts ============================================
platform linux -- Python 3.12.4, pytest-7.4.0, pluggy-1.5.0
rootdir: /home/example/Workspace/learning_precommit
plugins: anyio-4.4.0
collected 2 items                                                                                           

src/tests/test_main.py ..                                                                             [100%]

============================================= 2 passed in 0.30s =============================================
Enter fullscreen mode Exit fullscreen mode

Very good, we managed to create a simple application to apply the linter.

pre-commit

We have reached the long-awaited stage. Let's apply linters to our application using pre-commit.

Outside in the src directory, let's create a file called .pre-commit-config.yaml. This file will contain our hooks for each type of file we want to pass the linter to.

After creating the file, let's start adding the hooks:

Starting with Python, we will use flake8. You can choose other options, initially for this project I used this resource.

  • repos: Our group of repositories containing configurations for hooks.
  • repo: repository URL
  • rev: Β revision or tag to clone
  • hooks: Group of hooks
  • id: Identification for output at the terminal
  • additional_dependencies: Additional dependencies
  • language_version: Version of Python being used
  • exclude: A path or several to directories that will not be seen when executing the pre-commit, to use as an example, it was our alembic directory, which we do not want to be standardized, to remain the way I start or receive subsequent updates , for greater safety, I avoid putting the linter on it as a practice.
  • files: Location containing the files to be viewed by pre-commit, if not declared, the entire project will be viewed.
Flake8 
repos:
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
      additional_dependencies: [flake8==6.0.0]
      language_version: python3.9
      exclude: ^src/app/alembic
      files: ^src/
Enter fullscreen mode Exit fullscreen mode

GitHub logo PyCQA / flake8

flake8 is a python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code.

build status pre-commit.ci status Discord

Flake8

Flake8 is a wrapper around these tools:

  • PyFlakes
  • pycodestyle
  • Ned Batchelder's McCabe script

Flake8 runs all the tools by launching the single flake8 command It displays the warnings in a per-file, merged output.

It also adds a few features:

  • files that contain this line are skipped:

    # flake8: noqa

  • lines that contain a # noqa comment at the end will not issue warnings.

  • you can ignore specific errors on a line with # noqa: <error>, e.g., # noqa: E234. Multiple codes can be given, separated by comma. The noqa token is case insensitive, the colon before the list of codes is required otherwise the part after noqa is ignored

  • Git and Mercurial hooks

  • extendable through flake8.extension and flake8.formatting entry points

Quickstart

See our quickstart documentation for how to install and get started with Flake8.

Frequently Asked Questions

Flake8 maintains an FAQ in its documentation.

Questions or Feedback

…
View on GitHub

In the following hooks we have two differences, in the args, there are two ways that we can pass something that we want to ignore with the linter.

Hadolint

For Dockerfiles, we will use Hadolint. I've only been using it for a short time and I can't do without it anymore.

- repo: https://github.com/AleksaC/hadolint-py
  rev: v2.12.1b3
  hooks:
    - id: hadolint
    args: [--ignore, DL3008, --ignore , DL3007, --ignore, DL4006]
    exclude: ^src/app/alembic
Enter fullscreen mode Exit fullscreen mode

GitHub logo AleksaC / hadolint-py

Run hadolint in pre-commit without docker or system installation

hadolint-py

Add new versions Run tests License

A python package that provides a pip-installable hadolint binary.

The mechanism by which the binary is downloaded is basically copied from shellcheck-py.

Getting started

Installation

The package hasn't been published to PyPI yet, and may never be, as its primary purpose doesn't require it. However you can install it through git:

pip install git+https://github.com/AleksaC/hadolint-py.git@v2.12.1-beta
Enter fullscreen mode Exit fullscreen mode

To install another version simply replace the v2.12.0 with the version you want.

With pre-commit

This package was primarily built to provide a convenient way of running hadolint as a pre-commit hook, since haskell isn't supported by pre-commit. An alternative to this solution is to create a docker hook since hadolint provides a docker image, but I think that it has unnecessary amount of overhead.

Example .pre-commit-config.yaml with rules DL3025 and DL3018 excluded:

repos:
  - repo: https://github.com/AleksaC/hadolint-py
    rev: v2.12.1b3
    hooks:
      - id: hadolint
        args: [--ignore, DL3025, --ignore, DL3018]
Enter fullscreen mode Exit fullscreen mode
…
View on GitHub
Shellcheck

For shell-script files, quite useful, as development throughout the day when you add incorrect patterns in the script, this tool points out the points for standardization.

- repo: https://github.com/shellcheck-py/shellcheck-py
  rev: v0.10.0.1
  hooks:
    - id: shellcheck
      args:
      - --exclude=SC2046
      - --exclude=SC2006
      exclude: ^src/app/alembic
Enter fullscreen mode Exit fullscreen mode

GitHub logo shellcheck-py / shellcheck-py

python3/pip3 wrapper for installing shellcheck

build status pre-commit.ci status

shellcheck-py

A python wrapper to provide a pip-installable shellcheck binary.

Internally this package provides a convenient way to download the pre-built shellcheck binary for your particular platform.

installation

pip install shellcheck-py
Enter fullscreen mode Exit fullscreen mode

usage

After installation, the shellcheck binary should be available in your environment (or shellcheck.exe on windows).

As a pre-commit hook

See pre-commit for instructions

Sample .pre-commit-config.yaml:

-   repo: https://github.com/shellcheck-py/shellcheck-py
    rev: v0.10.0.1
    hooks:
    -   id: shellcheck
Enter fullscreen mode Exit fullscreen mode

View on GitHub


.pre-commit-config.yaml 
repos:
  - repo: https://github.com/pycqa/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
      additional_dependencies: [flake8==6.0.0]
      language_version: python3.9
      exclude: ^src/app/alembic
      files: ^src/
- repo: https://github.com/AleksaC/hadolint-py
  rev: v2.12.1b3
  hooks:
    - id: hadolint
    args: [--ignore, DL3008, --ignore , DL3007, --ignore, DL4006]
    exclude: ^src/app/alembic
- repo: https://github.com/shellcheck-py/shellcheck-py
  rev: v0.10.0.1
  hooks:
    - id: shellcheck
      args:
      - --exclude=SC2046
      - --exclude=SC2006
      exclude: ^src/app/alembic
Enter fullscreen mode Exit fullscreen mode

With the file defined, let's put the pre-commit to run. I'm going to do it a little differently than the documentation, you can consult the documentation for more information. If you can run pip install pre-commit and start using its commands and run the .pre-commit-config.yaml file.

The way I will use pre-commit will be using a Dockerfile. Let's now create a directory called containers, inside it we will add a file called Dockerfile. Then add the code below:

Dockerfile 
FROM python:3.9-slim

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y --no-install-recommends git && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /linter

RUN pip install --no-cache-dir pre-commit

COPY containers/ containers/
COPY scripts/ scripts/
COPY src/ src/
COPY .pre-commit-config.yaml .
COPY .git .git

CMD ["pre-commit", "run", "--all-files"]
Enter fullscreen mode Exit fullscreen mode

Notice in our Dockerfile that I add the command to install Git, this is necessary because pre-commit uses Git hooks. Next we will perform the pre-commit installation.

Now let's copy our project into the container, see that even the .pre-commit-config.yaml file is being copied, this will be necessary to execute our configurations inside the container.

Now let's create 2 shell-script files to run our container. Let's create a new directory called scripts. Inside it we will create two files build.sh and linter.sh.

scripts/build.sh

In this file we will add our build command for our Dockerfile.

#!/bin/bash
docker build . -f containers/Dockerfile -t project_linter:latest --rm
Enter fullscreen mode Exit fullscreen mode
scripts/linter.sh

Then in the linter.sh file we will add the following code. Note that I pass the docker commands within conditions, this way I can run the linter for each type of file, if the linter detects any correction, the script will stop running until it is adjusted, to continue with the next step.

#!/bin/bash
./scripts/build.sh
docker run --rm project_linter:latest pre-commit run --all-files flake8
status=$?
if test $status -ne 0
then
    exit $status
fi

docker run --rm project_linter:latest pre-commit run --all-files hadolint
status=$?
if test $status -ne 0
then
    exit $status
fi

docker run --rm project_linter:latest pre-commit run --all-files shellcheck
status=$?
if test $status -ne 0
then
    exit $status
fi
Enter fullscreen mode Exit fullscreen mode

Testing the linter

Now is the moment of truth.

Let's run our script file, ./scripts/linter.sh.

[+] Building 6.3s (14/14) FINISHED                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 424B                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                            1.1s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6  0.0s
 => [internal] load build context                                                                             0.0s
 => => transferring context: 11.66kB                                                                          0.0s
 => CACHED [2/9] RUN apt-get update &&     apt-get install -y --no-install-recommends git &&     rm -rf /var  0.0s
 => CACHED [3/9] WORKDIR /linter                                                                              0.0s
 => [4/9] RUN pip install --no-cache-dir pre-commit                                                           4.8s
 => [5/9] COPY containers/ containers/                                                                        0.0s 
 => [6/9] COPY scripts/ scripts/                                                                              0.0s 
 => [7/9] COPY src/ src/                                                                                      0.0s 
 => [8/9] COPY .pre-commit-config.yaml .                                                                      0.0s 
 => [9/9] COPY .git .git                                                                                      0.0s 
 => exporting to image                                                                                        0.2s 
 => => exporting layers                                                                                       0.1s
 => => writing image sha256:2005b56faa9f3ad9304712b10515d759a5e84ec553924f6b24d5f0be30d7e664                  0.0s
 => => naming to docker.io/library/project_linter:latest                                                      0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

src/app/main.py:5:1: E302 expected 2 blank lines, found 1
src/app/main.py:9:1: E302 expected 2 blank lines, found 1
src/app/main.py:11:40: W292 no newline at end of file
src/tests/test_main.py:6:1: E302 expected 2 blank lines, found 1
src/tests/test_main.py:11:1: E302 expected 2 blank lines, found 1
src/tests/test_main.py:14:59: W292 no newline at end of file
Enter fullscreen mode Exit fullscreen mode

Look what we have, we have several lines to be corrected. We will correct it in the next run again. Note below that our Flake8 linter has now passed. But our Dockerfile needs fixes, let's fix it then run it again.

[+] Building 0.7s (14/14) FINISHED                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 424B                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                            0.5s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6  0.0s
 => [internal] load build context                                                                             0.0s
 => => transferring context: 7.52kB                                                                           0.0s
 => CACHED [2/9] RUN apt-get update &&     apt-get install -y --no-install-recommends git &&     rm -rf /var  0.0s
 => CACHED [3/9] WORKDIR /linter                                                                              0.0s
 => CACHED [4/9] RUN pip install --no-cache-dir pre-commit                                                    0.0s
 => CACHED [5/9] COPY containers/ containers/                                                                 0.0s
 => CACHED [6/9] COPY scripts/ scripts/                                                                       0.0s
 => [7/9] COPY src/ src/                                                                                      0.0s
 => [8/9] COPY .pre-commit-config.yaml .                                                                      0.0s
 => [9/9] COPY .git .git                                                                                      0.0s
 => exporting to image                                                                                        0.0s
 => => exporting layers                                                                                       0.0s
 => => writing image sha256:605dd086f6a94f8d465092eaa8d027860358ce99e20bac4ec31a3e87199f5c7e                  0.0s
 => => naming to docker.io/library/project_linter:latest                                                      0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Hadolint.................................................................Failed
- hook id: hadolint
- exit code: 1

containers/Dockerfile:11 DL3013 warning: Pin versions in pip. Instead of `pip install <package>` use `pip install <package>==<version>` or `pip install --requirement <requirements file>`
Enter fullscreen mode Exit fullscreen mode

Corrections done successfully, all our linter passed.

[+] Building 0.7s (14/14) FINISHED                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                          0.0s
 => => transferring dockerfile: 431B                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                            0.5s
 => [internal] load .dockerignore                                                                             0.0s
 => => transferring context: 2B                                                                               0.0s
 => [1/9] FROM docker.io/library/python:3.9-slim@sha256:a6c12ec09f13df9d4b8b4e4d08678c1b212d89885be14b6c72b6  0.0s
 => [internal] load build context                                                                             0.0s
 => => transferring context: 7.14kB                                                                           0.0s
 => CACHED [2/9] RUN apt-get update &&     apt-get install -y --no-install-recommends git &&     rm -rf /var  0.0s
 => CACHED [3/9] WORKDIR /linter                                                                              0.0s
 => CACHED [4/9] RUN pip install --no-cache-dir pre-commit==3.5.0                                             0.0s
 => CACHED [5/9] COPY containers/ containers/                                                                 0.0s
 => CACHED [6/9] COPY scripts/ scripts/                                                                       0.0s
 => [7/9] COPY src/ src/                                                                                      0.0s
 => [8/9] COPY .pre-commit-config.yaml .                                                                      0.0s
 => [9/9] COPY .git .git                                                                                      0.0s
 => exporting to image                                                                                        0.0s
 => => exporting layers                                                                                       0.0s
 => => writing image sha256:1707d54754106d4eb8844d546a6b7a29b708ae2d1106e973d3d7d9b0e1c2b219                  0.0s
 => => naming to docker.io/library/project_linter:latest                                                      0.0s
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/pycqa/flake8.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
flake8...................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Hadolint.................................................................Passed
[INFO] Initializing environment for https://github.com/pycqa/flake8.
[INFO] Initializing environment for https://github.com/pycqa/flake8:flake8==6.0.0.
[INFO] Initializing environment for https://github.com/AleksaC/hadolint-py.
[INFO] Initializing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Installing environment for https://github.com/shellcheck-py/shellcheck-py.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
shellcheck...............................................................Passed
Enter fullscreen mode Exit fullscreen mode

About the author:

sc0v0ne image

sc0v0ne

Machine learning, deep learning, and raw code. Presented clearly and with examples.

A little more about me...

Graduated in Bachelor of Information Systems, in college I had contact with different technologies. Along the way, I took the Artificial Intelligence course, where I had my first contact with machine learning and Python. From this it became my passion to learn about this area. Today I work with machine learning and deep learning developing communication software. Along the way, I created a blog where I create some posts about subjects that I am studying and share them to help other users.

I'm currently learning TensorFlow and Computer Vision

Curiosity: I love coffee

My Latest Posts

sc0v0ne

Did you say Free GPU?

sc0v0ne ・ Feb 29

#python #gpu #ai #machinelearning
sc0v0ne

My Super Powers as a Software Developer - 2024

sc0v0ne ・ Jan 6

#tools #softwaredevelopment #workstations #productivity
sc0v0ne

Becoming Efficient with Pomodoro

sc0v0ne ・ Feb 12

#tools #productivity #softwaredevelopment

Favorites Projects Open Source

Top comments (0)

Read next

siddhantkcode profile image

Building smarter Docker Images for your development environments with Pants πŸš€

Siddhant Khare -

whiskeygerman profile image

Web canvas with generative neural networks with elements of visual programming.

German Whiskey -

codewith_ili profile image

How to Run MongoDB in Docker

Ili Aliaj -

kevin_menesesgonzlez profile image

How I Automated My Workflow by Connecting Python to Google Sheets API

Kevin Meneses GonzΓ‘lez -