sys.meta_path
is one of the best-kept secrets in the Python standard library, despite being squarely located in a module every Python programmer is familiar with. Some might say the secrecy is well-warranted, though; it's not usually a good idea to go messing around with tools and functions that may subvert programmer expectations the way this oddly-named list potentially can. But isn't "do not" better than "know not?" So before a quick primer on dependency injection, let's examine some of the ins and outs of sys.meta_path
.
Whenever you use an import
statement, say, import os
, that's functionally equivalent to the following code:
os = __import__("os")
It's a mild oversimplification, since stuff like submodules would require a little bit more work (as we'll have to deal with later, actually), but the main point to be made here is that the __import__
built-in does the heavy lifting under the hood for module imports in Python, and now we know that it exists, we can examine its control flow. First, it checks sys.modules
, a dict containing already-loaded modules, just in case we've already imported the requested module before. Next, it checks sys.path
- a list containing filesystem paths for Python to search through for potential modules; you probably know about this list already. However, if the sys.meta_path
list is not empty, than before looking through sys.path
, every meta importer hook in sys.meta_path
will be queried before the filesystem is checked, giving us an easy way to intercept module imports and do all sorts of freakishly wonderful things, like dependency injection.
So what is dependency injection? It's roughly an applied version of the dependency inversion principle, often for module-sized stuff. Say your application has several different front-ends, maybe targeting different platforms or UI toolkits. As long as you have a common interface, dependency injection would allow you to do something like:
user_info = info_getter.get_user_info()
frontend = dependency_injector.get("frontend")
frontend.display_user_info(user_info)
"So what? I can do that already with a Facade or something, maybe a config switch," I can hear you say. For very simple use cases, that may actually be better - simple is better than complex, remember? However, with dependency injection, we can easily provide module-level reuse, as well as significantly cut down on boilerplate. What if I told you, by the end of this article, you could write something like this:
import myapp.dependency.frontend as frontend
popup = frontend.Popup(popup_style=frontend.ERROR)
popup.display()
One import that works regardless of whether we're using GTK+, Qt, ncurses, or whatever your program needs to function. You might be thinking some very dark thoughts at this point. It's true, all sorts of dark magic could be performed depending on the details of our import hooks. But readability counts, so it's important that we not stray too far off the beaten path. So let's set some requirements for our dependency injection framework before writing our implementation:
- All dependency-injected import paths begin with
myapp.virtual
- Provided dependencies are known beforehand and registered with a
provide
method
All the magic - and there's not all that much - relies almost entirely on creating a meta path finder and adding it to sys.meta_path
. A meta path finder is a very simple object with a single public method, find_spec
:
import importlib.abc
import importlib.machinery
class DependencyInjectorFinder(importlib.abc.MetaPathFinder):
def __init__(self, loader):
# we'll write the loader in a minute, hang tight
self._loader = loader
def find_spec(self, fullname, path, target=None):
"""Attempt to locate the requested module
fullname is the fully-qualified name of the module,
path is set to __path__ for sub-modules/packages, or None otherwise.
target can be a module object, but is unused in this example.
"""
if self._loader.provides(fullname):
return self._gen_spec(fullname)
def _gen_spec(self, fullname):
spec = importlib.machinery.ModuleSpec(fullname, self._loader)
return spec
# we'll also add it to sys.meta_path later
If a meta path finder provides the requested module, then it should return an instance of the importlib.machinery.ModuleSpec
class, which is a fairly simple affair with a small handful of attributes that lets Python's import machinery know what it needs to know to take the next steps in importing the module the user requested. For our purposes, we're interested in two attributes (the only required ones): ModuleSpec.name
, which is the name of the requested module, and ModuleSpec.loader
, which is the loader object that Python should use to actually load the module - you'll notice the self._loader
lines above that reference a loader object, as well. A loader object is a very simple class with two required methods in modern Python (3.4 onwards): create_module
, which takes a ModuleSpec
as its sole argument and returns an object that Python will consider to be the new module, and exec_module
, which takes the new module as its sole argument and executes it. So a no-op, barebones loader looks something like this:
class Loader(importlib.abc.Loader):
def create_module(self, spec):
raise NotImplementedError
def exec_module(self, module):
raise NotImplementedError
In addition to those two methods, we should implement the provide
method we talked about earlier to signal that a certain dependency is provided, in addition to the provides
method that our finder referenced earlier, that indicates whether the requested module is part of our dependency injection framework. Here's the implementation:
import importlib.abc
import sys
import types
class DependencyInjectorLoader(importlib.abc.Loader):
_COMMON_PREFIX = "myapp.virtual."
def __init__(self):
self._services = {}
# create a dummy module to return when Python attempts to import
# myapp and myapp.virtual, the :-1 removes the last "." for
# aesthetic reasons :)
self._dummy_module = types.ModuleType(self._COMMON_PREFIX[:-1])
# set __path__ so Python believes our dummy module is a package
# this is important, since otherwise Python will believe our
# dummy module can have no submodules
self._dummy_module.__path__ = []
def provide(self, service_name, module):
"""Register a service as provided via the given module
A service is any Python object in this context - an imported module,
a class, etc."""
self._services[service_name] = module
def provides(self, fullname):
if self._truncate_name(fullname) in self._services:
return True
else:
# this checks if we should return the dummy module,
# since this evaluates to True when importing myapp and
# myapp.virtual
return self._COMMON_PREFIX.startswith(fullname)
def create_module(self, spec):
"""Create the given module from the supplied module spec
Under the hood, this module returns a service or a dummy module,
depending on whether Python is still importing one of the names listed
in _COMMON_PREFIX.
"""
service_name = self._truncate_name(spec.name)
if service_name not in self._services:
# return our dummy module since at this point we're loading
# *something* along the lines of "myapp.virtual" that's not
# a service
return self._dummy_module
module = self._services[service_name]
return module
def exec_module(self, module):
"""Execute the given module in its own namespace
This method is required to be present by importlib.abc.Loader,
but since we know our module object is already fully-formed,
this method merely no-ops.
"""
pass
def _truncate_name(self, fullname):
"""Strip off _COMMON_PREFIX from the given module name
Convenience method when checking if a service is provided.
"""
return fullname[len(self._COMMON_PREFIX):]
There's seemingly a lot of code here at first glance (even though most of it is comments!), so let's walk through this class' methods. Up first are the provide
and provides
methods that we've talked about earlier; there's not a whole lot of magic present in either one. create_module
returns a so-called dummy module if we're trying to import either myapp
or myapp.virtual
, and there's a good reason for that. Say we have the following line of code:
import myapp.virtual.frontend
Under the hood, this generates three distinct searches in Python's import machinery: One for myapp
, another for myapp.virtual
, and lastly one for myapp.virtual.frontend
. Since obviously myapp
and myapp.virtual
don't actually exist anywhere on the system, but Python will complain if they aren't loaded, we claim we provide both - note how provides
will return True
if queried for both myapp
and myapp.virtual
and return an empty dummy module to placate Python's import machinery. Only when we encounter myapp.virtual.frontend
do we check against the list of provided dependency-injection services.
At this point, we've actually written all the plumbing we need to, so let's wrap up and see how we'd use these classes in practice:
import sys
class DependencyInjector:
"""
Convenience wrapper for DependencyInjectorLoader and DependencyInjectorFinder.
"""
def __init__(self):
self._loader = DependencyInjectorLoader()
self._finder = DependencyInjectorFinder(self._loader)
def install(self):
sys.meta_path.append(self._finder)
def provide(self, service_name, module):
self._loader.provide(service_name, module)
class FrontendModule:
class Popup:
def __init__(self, message):
self._message = message
def display(self):
print("Popup:", self._message)
if __name__ == "__main__":
injector = DependencyInjector()
# we could install() and then provide() if we wanted, order isn't
# important for the below two calls
injector.provide("frontend", FrontendModule())
injector.install()
# note that these last three lines could exist in any other source file,
# as long as injector.install() was called somewhere first
import myapp.virtual.frontend as frontend
popup = frontend.Popup("Hello World!")
popup.display()
At this point, it'd be worth it to take a step back and examine at a high level what is going on behind the scenes:
- The call to
injector.install
appends ourDependencyInjectorFinder
onto thesys.meta_path
list, so Python will use it for future imports. - The line
import myapp.virtual.frontend as frontend
triggers three module searches, as mentioned earlier - one formyapp
, thenmyapp.virtual
, thenmyapp.virtual.frontend
:- Python combs through
sys.meta_path
, looking for meta path finders. If your system Python is anything like mine, ourDependencyInjectorFinder
will be the only one not defined by_frozen_importlib
, which is a part of Python itself. - For each found meta path finder, Python queries the finder's
find_spec
method, seeing if the finder provides the given module. - Obviously
myapp.virtual.frontend
doesn't exist on the filesystem, so it falls to our meta path finder to handle it. In all three cases, we return aModuleSpec
instance withname
set to the same name that Python asked if we could find, andloader
set to our customDependencyInjectorLoader
. - Next, Python will call the
create_module
method on our loader for theModuleSpec
in question. Formyapp
andmyapp.virtual
, our loader recognizes that these are dummy modules and returns the same dummy module for both (tricking Python into believing the module was loaded), but returns the instance ofFrontendModule
we gave it withinjector.provide()
upon being asked to loadmyapp.virtual.frontend
. Python allows any valid object to function as a module, so a plain old class is perfectly fine to return. - Python will finally call the
exec_module
method on our loader, passing it the object we returned fromcreate_module
. Python requires theexec_module
method to be present but doesn't really care about its behavior; normally, the method would execute the module's code in a new namespace, but since we already have fully-formed modules ready to go - either our pre-made dummy module or ourFrontendModule
instance - we simply do nothing insideexec_module
. - Wash, rinse, and repeat for each successive module search.
- Python combs through
- After all that,
frontend.Popup
is functionally the same thing asFrontendModule.Popup
, and the rest, as they say, is history.
As I mentioned earlier in the article, it's important not to abuse language features like this - with great power comes great responsibility, after all - and this especially rings true when it comes to metaprogramming. Still, there's a time and place for everything, import hooks included; but please use them responsibly.
Finally, the complete code example is up as a gist on GitHub if you'd like something you can download and pore over. Happy hooking!
Top comments (3)
Hi Kermit, super interesting. I didn't know about sys.meta_path - seems also very handy to develop plugin systems like you did! Great for a big customizable app
A small suggestion: if you re-edit your post you can add syntax highlighting with tags like this:
You just need to add
python
at the end of the three beginning ticks, like this:Ah, appreciate the tip on syntax highlighting! I've updated the article accordingly.
Interesting post! Thanks.
Have you tried seeing how static type checking behaves with this meta_path trick? Or would IDEs complain that the myapp.virtual.foo module does not exist?