Demystifying Python metaclasses Why are they so special?

Classes with class — metaclasses in Python

Hi! My name is Jakub Barański and I’m a full stack developer at Profil Software, a Python software development company that also offers the services of react native agency and more. For creating backend, I usually use Python, so I wanted to dive into one of the aspects of this language that I find very interesting.

When you are beginning to learn any object oriented programming language, you need to understand two concepts: classes and instances. A basic simplification that allows you to quickly understand them is that classes are blueprints or schematics, logical entities based on which instances are created. On the other hand, instances are actual “physical” objects that have a state and some specific behavior. Going further into machine-related stuff, declaring a class will not allocate memory, whereas creating an instance will. This is a bit of an oversimplification, but for the most part it is correct.

Instances are created using a constructor, a block of code that specifies what should happen when memory is allocated for an object. Things that in other programming languages you would do in a constructor, you usually do in the init method in Python.

class Dog:
    def __init__(self, good_boy: bool):
        self.good_boy = good_boy

However, when you take a close look at arguments passed to the constructor, you will notice that the first argument is always self. Isn’t this object already created at this point? Well…it is. What you will most likely find when googling “python constructor” is not actually a constructor but a so-called “initializer”. It is not responsible for creating an object instance but for instantiating its state. The method that creates an object in Python is called new .

class Dog:
    def __new__(cls, *args, **kwargs):
        return super().__new__(cls, *args, **kwargs)

Enjoying this article? Need help? Talk to us!
Receive a free consultation on your software project

Let’s take a look at the arguments again. This time we have cls as the first argument instead of self. Its name can probably already suggest what it is, but let’s do a sanity check by adding print to the constructor.

Now when we create our object we should see what that mysterious cls is all about.

dog = Dog()
# <class 'Dog'>

The actual class is passed to the constructor of the object. But should that even be possible? Didn’t we establish in the first paragraph that typical classes are just logical entities that are not stored in runtime memory? Well, of course we did. Using the built-in id() method, however, will return the memory address of our class. Therefore, we can now say with absolute certainty that is an actual object. This is an example of a first class citizen — an entity that can be used in any operation in your code, whether that’s passing it as an argument to the function, returning it from the function, or even modifying it at runtime. We often say that in Python everything is an object, but for some reason it’s hard to accept that classes are objects too.

OK, but if a class is an object, shouldn’t it also be constructed somehow? Shouldn’t it have some…well, class? Thanks to Python’s great ability for introspection, we can easily check that.

dog_class = dog.__class__
print(dog_class)
# <class 'Dog'>
print(dog_class.__class__)
# <class 'type'>

Now it will get a little weird. Looks like the class of our Dog class is type It may look very similar to the built-in method type, that we use to acquire the type of an object.

type('hello')
# <class 'str'>

It’s very similar because it’s actually the very same thing. It might be getting a bit confusing at this point, but don’t worry. When we are in trouble, Python has another useful built-in method for us to clear things up.

help(type)
"""
Help on class type in module builtins:

class type(object)
 | type(object_or_name, bases, dict)
 | type(object) -> the object's type
 | type(name, bases, dict) -> a new type
...
"""

So as you can see, the built-in type method is overloaded and behaves differently depending on what it gets as arguments (it doesn’t really comply with the Zen of Python, right? “Simple is better than complex”?. Oh well, they had their reasons).

Keeping that in mind, we’ve slowly but surely reached the actual subject of this article — metaclasses. Our built-in type method is a metaclass - a class that is used as a blueprint to create classes. Let’s try to go deeper. What is a class of a metaclass?

obj.__class__.__class__.__class__
# <class 'type'>

Fortunately, that’s it. All classes in Python by default have type as a metaclass. A metaclass is then used to create a class that is then used to create objects.

But what can we use this knowledge for? Firstly, we can use the second functionality of type to do something that is unheard of in most other programming languages. We can dynamically create classes. We can invoke type and provide it with three arguments. First is the name of our new class. Second is a tuple of bases — parent classes that our new class should be derived from. Third is a dictionary with our class attributes.

Cat = type("Cat", (), {})
c = Cat()
c
# <__main__.Cat object at 0x7fcbb21eb358>

Creating classes like that is not a very common scenario, but it can be useful in some cases.

Also, now that we know type is a metaclass, we can subclass it to create our own metaclasses and provide our own logic to the class constructor.

class GreatestMetaclass(type):
    def __new__(metacls, name, bases, attrs):
        x = super().__new__(metacls, name, bases, attrs)
        # do some absolutely amazing stuff here...
        return x

Now if we want to create a class that implements our GreatestMetaclass we can do it like this:

class GreatestClass(meta=GreatestMetaclass):
    pass

GreatestClass will now implement all of the magic we wrote in the metaclass definition.

What can we use a metaclass for? Plenty of things. For example, you can use it to keep a registry of classes that implements it.

registry = {}


class RegistryMetaclass(type):
    def __new__(metacls, name, bases, attrs):
        registry[name] = super().__new__(metacls, name, bases, attrs)
        return registry[name]

Interested in working for our software development company in Poland?
Check out our current job offers now

Metaclasses are also used a lot in many different Python frameworks. We can take a look at the biggest Python web framework Django and its extensive use of metaclasses in its ORM (the custom logic of just the ModelBase metaclass constructor takes about 300 lines of code!).

Thanks to the use of the ModelBase metaclass in Model classes that are used to reflect database tables, we can simply declare our model like this:

class Person(models.Model):
    first_name = models.CharField(max_length=30)
    last_name = models.CharField(max_length=30)

With such an implementation, we can very quickly create models with a minimal amount of code. We just declare fields by creating attributes and assigning field types to them. All the work related to connecting your class with Django ORM is covered by the metaclass’s inner mechanisms and you can instead focus on creating your model structure and application logic.

Another example of a popular library where metaclasses are used is Django Rest Framework and it's serializers. SerializerMetaclass creates a _declared_fields dictionary in the serializer class which contains all instances of the Field class that were included in the serializer class as attributes (or as attributes in the inherited superclass). Objects that implement SerializerMetaclass can then create a deep copy of these fields and use them.

class UserSerializer(serializers.Serializer):
    email = serializers.EmailField()
    username = serializers.CharField(max_length=200)

One metaclass that many python programmers might have used is an abstract base class metaclass (ABCMeta). Usually we create abstract classes in Python by deriving our class from the ABC, but in reality it is just a helper class that has ABCMeta as its metaclass. ABC was created to bypass the usage of metaclasses in cases that can be more confusing than simple inheritance.

from abc import ABC, ABCMeta


class MyABC(ABC):
    pass


# this is the same thing as :


class AlsoABC(metaclass=ABCMeta):
    pass

Ok, so earlier I wrote that “there are plenty of things you can use metaclasses for”. The real question is though — SHOULD we do this? To be completely honest, I had only a handful of situations where I thought that creating a metaclass would help me with anything. Even then I immediately came to the conclusion that there are a ton of other solutions that are much simpler. I think it’s useful to know they exist because it allows you to better understand how Python is designed and how it works under the hood. However, if you want to actually use these features in your code you should have a good reason to do so. Otherwise, you are just making your implementation more obscure and that is never a good idea. Once again, look at the Zen of Python: “If the implementation is hard to explain, it’s a bad idea”. The creator of said zen has an opinion about metaclasses, so in the end let me just quote him:

Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t.
-Tim Peters