Make your Python program highly configurable

The more complex a program is, the more parameters it tends to accept in order to tune it's behavior. A configurable program is easier to adapt to the needs of the user (and your own) and reduces the amount of code you have to modify if you want to change certain things.

There are three ways to specify parameters in your code:

1. Directly in the source code.
2. Inside a separate configuration file.
3. With the arguments from the command line.

Each one should override the parameters of the previous one. How do we implement this in a clean way?

1. Get all the parameters separately

From the source code

This needs no explanation. The default configuration is hard-coded.

default_config = {
    "mode": "demo",
    "timeout": 5,
    "color": blue,
    "language": "en"
}

From a configuration file

There are countless ways to read a configuration file, from a custom parsing function to a library like configparser. Obviously, your choice will determine the format of the file and each one has its own pros and cons.

To keep this simple, I will use a regular JSON file:

import json

with open("config.json", "r") as f:
    file_config = json.load(f)

From the command line

Again, there are multiple ways to parse the arguments from the command line. I think the most extended is argparse, because it is flexible and implements directly a lot of commonly expected behaviour, like an autogenerated help message and more.

However, it makes the code too long for an example like this, so I'll use a custom piece of code to parse arguments with the form --key=value.

import sys

params = filter(lambda x: x[:2] == "--", sys.argv)
params = map(lambda x: x.split("="), params)
cli_config = {k[2:]: v for [k,v] in params}

2. Override values present in lower levels

In Python, the double star (**) operator is used to unpack dictionaries. It extracts the key-value pairs to be used elsewhere.

One of the most interesting uses of this operator is to merge dictionaries, building a new one from other unpacked. As they only allow unique keys, the dictionaries unpacked on the right replace the values repeated on the left.

final_config = {**default_config, **file_config, **cli_config}

Example

Let's put together what we have:

main.py

import sys
import json

## default
default_config = {
    "mode": "demo",
    "timeout": 5,
    "color": "blue",
    "language": "en"
}

print("default", default_config)

## file
with open("config.json", "r") as f:
    file_config = json.load(f)

print("file   ", file_config)

## command line
params = filter(lambda x: x[:2] == "--", sys.argv)
params = map(lambda x: x.split("="), params)
cli_config = {k[2:]: v for [k,v] in params}

print("cli    ", cli_config)

## merge them
final_config = {**default_config, **file_config, **cli_config}

print("final  ", final_config)

config.json

{
    "color": "red",
    "timeout": 10
}

Output

$ python main.py --color=green --language=es
default {'mode': 'demo', 'timeout': 5, 'color': 'blue', 'language': 'en'}
file    {'color': 'red', 'timeout': 10}
cli     {'color': 'green', 'language': 'es'}
final   {'mode': 'demo', 'timeout': 10, 'color': 'green', 'language': 'es'}

Conclusion

Making your programs configurable will make a lot of your work easier. Allowing the user (and yourself as a developer) to customize parts of your application without modifying the source code both in a persistent and a dynamic way always makes a difference.

Recommended reading

How to make an ethical crawler in Python

MiguelMJ ・ Dec 27 '20

#python #tutorial #webscraping #web

I made a recommendation system for Stack Overflow unanswered questions

MiguelMJ ・ Jan 15 '21

#showdev #python #project

You can follow me on Twitter! 🐦