Flask / Quart - manage module loading and splitting

Working with Flask or the fabulous Quart in Python, you sometime want to "split" your routes (or views) in several packages. That's something very common in Go, but a bit more complex with Python. I've got something to propose.

The reason is how Python defined a package. With Go, a package is a set of files that are all injected once. But with Python, it's a set of module that are independent.

So, the time has come when you begin to use Blueprints. A very nice way to write sub-routes that you can classify, prefix and activate.

That's ok, let's see the problem.

Note, I use Quart but you can change the entire examples to use Flask (remove async and change imports)

Classification of routes

Let's imagine we want to create this application structure:

app.py
api/
    __init__.py
    auth.py
    content.py

The app.py will load the api package to register it as a Blueprint. That's common, and we will not touch this file anymore. So, write the app.py file like this:

""" The main application """

from quart import Quart

from api import api

app = Quart(__name__)
app.register_blueprint(api, url_prefix="/api")

if __name__ == "__main__":
    app.run(debug=True)

Manage the routes in modules

Now, let's take a look at the routes package. And see the problem happening.
First, declare a Blueprint in api/__init__.py

# api/__init__.py
from quart import Blueprint

api = Blueprint('api', __name__)

The api variable is global and can be imported inside any modules.
That's not a bad idea.

# in api/auth.py
from . import api

@api.route("/login")
async def login():
    return "Welcome"

@api.route("/logout")
async def logout():
    return "Bye !"

And...

# in api/content.py
from . import api

@api.route("/")
async def index():
   return "The content !"

Back to __init__.py, how to import the modules?

There are many difficulties...

• You cannot import auth and content modules before the api declaration, because the module is not yet initialized (keep in mind that api should be initialized to use @api.route decorator...
• if you put imports after the api declaration, the linter will complain about the placement of imports
• and moreover, if you import modules without any use of them, so the linter will complain (and it is right, it's bad, ugly)

So, let's make a "programmatic import"! A good idea, isn't it?

# in api/__init__.py

from quart import Blueprint

api = Blueprint('api', __name__)

for module in ["auth", "content"]:
    __import__(f"{__name__}.{module}")

This is not that bad actually, everything is working as expected.

Yes, but... What if we forget a module or if we make a mistake in the name? And what if I move, rename or delete a module ?

Make it a few more automatic

We need to detect the modules inside the package. And, believe it or not, that's not so easy. We could, for example, use os.walk or glob.glob to find files. But there is a complication: a module can be "compiled" as a library (.so file), and there are many other extensions to detect. By chance, we have a solution.

There is a __loader__ inside each package. A lot of (even good) Python developers ignore this, and that's a pity. Because this variable contains a very useful SourceFileLoader that provides a FileReader. This is a generator that gives a solution to retrieve the modules inside the package.

Ho !

So! It's time to make a nice and tricky loop!

# app/__init__.py
from importlib import import_module
from pathlib import Path

from quart import Blueprint

api = Blueprint('api', __name__)

for mod in __loader__.get_resource_reader().contents():
    mod = Path(mod).stem
    if mod == "__init__":
        continue
    import_module(__package__ + "." + mod, __package__)

But... one more time, a problem may happen.

What if another file than a Python source file (or precompiled file) exists in the module ? For example, a README.md file.

There are plenty of ways to check this, for example having a list of possible extensions.

But, are you sure to know the entire list of Python resource extensions?
I mean, you think about .py and .pyc - but do you remember that the .pyo extention exists? And are they other forgotten names?

So, what I love to do is to guess the mimetype.

# app/__init__.py
import mimetypes
from importlib import import_module
from pathlib import Path

from quart import Blueprint

api = Blueprint('api', __name__)

for mod in __loader__.get_resource_reader().contents():
    if "python" not in str(mimetypes.guess_type(mod)[0]):
        continue

    mod = Path(mod).stem
    if mod == "__init__":
        continue

    import_module(__package__ + "." + mod, __package__)

And there we are!

Time to make it a "tool"

OK, copying this loop inside each package __init__ file is a waste of time, and it is not comfortable if we need to fix the behavior in case of bug.

Anyway, when you need to copy a portion of code, so it should be a function.

Note, the __loader__ is a SourceFileLoader which is a bit tricky to find in modules, and not easy at all to use. So, I prefer to make a "representation" of the class I need.

It's time to create a "tool". Create a "utils" package at the root of the project, and make the __init__.py file:

# utils/__init__.py

import mimetypes
from importlib import import_module
from pathlib import Path
from typing import Callable


class SourceFileLoader:
    """ Represents a SouceFileLoader (__loader__)"""
    name: str
    get_resource_reader: Callable

def load_modules(loader: SourceFileLoader):
    """Load the entire modules from a SourceFileLoader (__loader__)"""
    pkg = loader.name
    for l in loader.get_resource_reader().contents():

        if "python" not in str(mimetypes.guess_type(l)[0]):
            continue

        mod = Path(l).stem
        if mod == "__init__":
            continue

        import_module(pkg + "." + mod, pkg)

In the api and others packages, I can now do:

# api/__init__.py

from quart import Blueprint
from utils import load_modules

api = Blueprint("api", __name__)

load_modules(__loader__)

And now, that's fantastic! Our all modules are now injected from the package. I don't forget any file, everything is automated by a simple call to load_modules().

I know...

Yes, Python is an awesome language, easy to use and Flask or Quart are wonderful. But, as you can see, sometimes you need to make some tricky things to make your project more concise.

Here, to make this loader tool, we needed to know some deep knowledge of the language. But that's also something I really love when I use a technology: not to be happy of the basics. I like to use pdb to observe and discover what's behind the scene. Using dir() on a object, checking types...

So, this loader is a little example of what we can do to automate development and to make less code in our project.

I hope you enjoyed my article and that may help you 😄