Loading config files in Python

Config files are everywhere. There are lots of reasons your app might need to have one:

• You have configuration that you want to persist beyond a reboot.

• Your configuration represents a physical state; for example, it contains the settings for peripheral devices, a stored procedure for accomplishing a task, or maybe it expresses the layout of the live user interface.

• Your app's configuration cannot be easily expressed as a series of variables. CI pipelines, workflows, etc. feature a lot of complex nesting, repeated blocks, and even internal linking.

• You want the app to be able to persist its own changes to configuration, like changing of windows sizes, menu settings, or credentials. In this case, the config file is functioning more as a database than something the user writes.

In all of these cases, the structure of the config is very important and likely long-lived. Mistakes in your config syntax will be hard to undo, so it pays to have a plan upfront, and design for it to be extended and documented.

In this article, we'll learn how to load YAML config files in a way that is clean, easy to support, and easy to extend. We'll do this by creating our own YAML task automation syntax, which we'll call taskbook files:

# taskbook.yml

group: # name of group

tasks: # list of tasks
  - name: # name of task
    module: # module to use
    options:
      # key / value options

  # ...

We'll write a program to read them, which we'll call Taskable *. When finished, it will be easy to determine fields that are supported, validate config values safely, add more fields for future needs, and even access config values within our program as properties.

*Any similarity to Ansible playbook syntax, real or imagined, is purely coincidental. 😂

Create a command line tool

Let's create a file called taskable.py, to contain our implementation of Taskable:

# taskable.py
import argparse

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()

if __name__ == " __main__":
    main()

This provides the scaffolding for an argparse command line interface (for more info, see our article on Python CLIs).

You can run the script as follows:

python3 taskable.py

$ python3 taskable.py
usage: taskable.py [-h] file
taskable.py: error: the following arguments are required: file

To be able to read in a file, we need to create the file first, which we'll do next.

Create a taskbook file

I'll be using YAML for the config files because it's easy to read and I'm comfortable with it, but you can easily support JSON or TOML, as they offer similar APIs.

Create a taskbook.yml file and add the following:

# taskbook.yml
group: localhost
tasks:
  - name: copy file.txt to the place
    module: saucy.copy
    options:
      source: file.txt
      dest: /etc/file.txt

  - name: install a package
    module: cheesy.package
    options:
      name:
        - fzf
        - tree
      upgrade: true

  - name: enable the service
    module: lettuce.service
    options:
      enable: true
      start: true

At this point, we'll be able to run the following:

python3 taskable.py taskbook.yaml

However, nothing will happen because our app doesn't print anything yet.

Read in the YAML file

YAML files are easy to read with Python. There are multiple libraries available, but pyyaml is the de facto standard and is often installed on whatever system you're already on.

If you don't have pyyaml (or you're using a virtual environment because you're awesome), install it now:

pip install pyyaml

Then, in your taskable.py file, import the yaml package and read in the YAML file:

import yaml
...
data = yaml.safe_load(args.file)

Our taskable.py file so far:

# taskable.py
import argparse
import yaml

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)

if __name__ == " __main__":
    main()

At this point, you will be able to read in the YAML file, but there's still no output just yet. We could stop here and access its values as nested dictionaries and arrays, like so:

data["tasks"][0]["module"]

...but there are a couple problems with this.

First, there's no validation at all, so a malformed config file has unpredictable results. Second, strings are opaque data, so IDE auto-completion won't work; changing a field name will require manually searching through the code to do so; and I hope you never misspell a field name.

No, we can do a lot better, and we will, starting by building a model of our data in the next section.

Create the data model

We need a way to express our data format so that it's functional. For this purpose, I prefer to use attrs, which gives us data validation, makes our classes more performant, allows us to access our fields as properties with dramatically less boilerplate, and more.

Let's install attrs:

pip install attrs

Then add the following to your taskable.py file:

from typing import Any
...
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)

if __name__ == " __main__":
    main()

These two classes—Task and Taskbook—fully express the taskbook format. We won't instantiate them ourselves though, because we'll learn a method to do so automagically in the next section.

Structurize into models

"Structurize" is a $6 word (that I may have made up) that translates to, "load all your data into fancy model classes." I'm using it because "de-serialize" sounds awful and is harder to type. 😝

The easiest way to structurize your YAML data into attrs classes is by using the cattrs package. The simplest usage looks like this:

import cattrs
taskbook = cattrs.structure(data, Taskbook)

Let's add it to our taskable.py file:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)

if __name__ == " __main__":
    main()

That's all you need! cattrs will load the data into attrs classes after only being given the expected top-level class, which is Taskbook here.

If you need to tweak the behavior, cattrs provides a hook mechanism. It's a bit cumbersome, but it's easier than writing all the structurization code from scratch.

In the next section, we'll work on doing something useful with our data.

Use the data

At this point, we've fully structurized our data into classes, which means we can access our config data like this:

taskbook.tasks[0].module

This makes our code much easier to read and work with. Now we'll try using it to do stuff.

"Run" tasks

What good is our script if it can't run tasks? Let's add something to simulate "running" our hypothetical tasks, by adding the following to our taskable.pyfile:

...
print("group", taskbook.group)
for task in taskbook.tasks:
    print(f"run {task.module}: {task.name}")
...

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)
    print("group", taskbook.group)
    for task in taskbook.tasks:
        print(f"run {task.module}: {task.name}")

if __name__ == " __main__":
    main()

Running our hypothetical tasks will output the following:

python3 taskable.py taskbook.yml

$ python3 taskable.py taskbook.yml
group localhost
run saucy.copy: copy file.txt to the place
run cheesy.package: install a package
run lettuce.service: enable the service

It's not hard to imagine connecting this skeleton to real module implementations to drive real task execution.

List used modules

Maybe we'd like to inspect our taskbook to find out what modules it uses. This would be useful, for example, to install necessary modules before running our tasks.

Let's add a -l / --list option to list used modules and exit without running the tasks:

...
parser.add_argument("-l", "--list", action="store_true")
...
if args.list:
    used_modules = sorted(list(set(task.module for task in taskbook.tasks)))
    for module in used_modules:
        print(module)
    return
...

Our taskable.py file so far:

# taskable.py
import argparse
from typing import Any

import cattrs
import yaml
from attrs import define, field

@define
class Task:
    name: str
    module: str
    options: dict[str, Any] = field(factory=dict)

@define
class Taskbook:
    group: str
    tasks: list[Task]

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("-l", "--list", action="store_true")
    parser.add_argument("file", type=argparse.FileType("r"))
    args = parser.parse_args()
    data = yaml.safe_load(args.file)
    taskbook = cattrs.structure(data, Taskbook)
    if args.list:
        used_modules = sorted(list(set(task.module for task in taskbook.tasks)))
        for module in used_modules:
            print(module)
        return
    print("group", taskbook.group)
    for task in taskbook.tasks:
        print(f"run {task.module}: {task.name}")

if __name__ == " __main__":
    main()

Running taskable.py with the list mode enabled:

python3 taskable.py -l taskbook.yaml

$ python3 taskable.py -l taskbook.yaml
cheesy.package
lettuce.service
saucy.copy

Woot! Static analysis! And it was easy to implement because our data model is so well-defined.

Summary

In this tutorial, we've built up a versatile config loading mechanism.

This setup works equally well for tiny command line utilities as it does for large and complex data formats files like task workflows, specifications, and so on. You can continue growing your application by adding new fields and new data models, and avoid the malignant technical debt that springs from a muddled early config implementation.

The best part? Your configuration will be stable and serve the bedrock and foundation of your application, now and in the future. In the words of Eric S. Raymond:

Smart data structures and dumb code works a lot better than the other way around.

— Eric S. Raymond

Stay smart, people! 😄