Make your code more Pythonic with Magic Methods

Dunders

If you spend some time with Python you will eventually hear about something called “dunder”. Even if you haven’t and this name sounds odd to you I can bet that you’ve used it multiple times and just didn’t know what was called. To make it easier — dunder is an abbreviation of two words: “double under”, where “under” is yet another abbreviation, this time of the word “underscore”. Knowing this, you probably already know what I am talking about. Yes, the Python init method is one of many dunders (also sometimes called “magic method”) that we can use in Python.

Dunders are internal cogs that do work when on the surface we use built-in methods, operators and keywords that make Python so easy to read and write. For a good start, we can look at which dunders are provided with the base object that every other class inherits.

o = object()
dir(o)
[
    "__class__",
    "__delattr__",
    "__doc__",
    "__format__",
    "__getattribute__",
    "__hash__",
    "__init__",
    "__new__",
    "__reduce__",
    "__reduce_ex__",
    "__repr__",
    "__setattr__",
    "__sizeof__",
    "__str__",
    "__subclasshook__",
]

Straight off the bat we have 15 dunders. Let’s just take some of them and see what they do. If you’ve ever used Django you are probably familiar with the str method, used often when declaring models. This is the one responsible for returning the string representation of… well, anything in Python. Whenever you use the built-in function str(), this dunder will be called and it will decide what string will be returned. Similarly, when you use print to log something in the console, the str dunder will be called.

Some dunders are called when you use specific keywords in Python. For example, delattr is called when you use the keyword del, as when deleting dictionary entries for instance.

dictionary = { 'key': value }
del dictionary['key']

Of course, these 15 dunders are just a small fraction of what can be used. Some of the more useful are ones that allow us to override operators. This is a concept known for a very long time in other programming languages. In C++ you can overload operators to have a special meaning when used with objects that provide the overload definition. In Python you can do the same by implementing, for example, the add method that is responsible for handling the addition operator. To demonstrate, we can create a simple class that will allow us to “simplify” adding elements to a list.

class Appender:
    def __init__(self):
        self.array = []

    def __add__(self, other):
        self.array.append(other)
        return self

    def __str__(self):
        return str(self.array)


ap = Appender()
ap + 3 + 5 + 6 + 9 + 12

print(ap)
# output: [3, 5, 6, 9, 12]

So, we implemented the Appender class that overrides the add method to append a new element to a list it keeps inside every time we use a + operator. After that we return the Appender instance so we can do it many times over by chaining operators.

Enjoying this article? Need help? Talk to us!
Receive a free consultation on your software project

How about something useful?

Ok, so Appender shows that you can do some creative stuff with operators in Python to cause non-standard behavior of the code. Some of it is really useful and is even in the Python standard library. For example, have you ever wondered how you add timedelta to the datetime object? Yup, operator overload. Here is the example from the cpython repository, file datetime.py.

def __add__(self, other):
    "Add a datetime and a timedelta."
    if not isinstance(other, timedelta):
        return NotImplemented

    delta = timedelta(
        self.toordinal(),
        hours=self._hour,
        minutes=self._minute,
        seconds=self._second,
        microseconds=self._microsecond,
    )
    delta += other
    hour, rem = divmod(delta.seconds, 3600)
    minute, second = divmod(rem, 60)
    if 0 < delta.days <= _MAXORDINAL:
        return type(self).combine(
            date.fromordinal(delta.days),
            time(hour, minute, second, delta.microseconds, tzinfo=self._tzinfo),
        )
    raise OverflowError("result out of range")

One of the interesting approaches of using dunders can also be found in pathlib. It is a library meant to help you work with filesystem. Before it was included in the standard library (it’s been there since Python 3.4) you would most likely have used os.path for creating path names and opening files. However, if you did it even once you know how cumbersome that was. Say you want to get the parent directory of the current code file. You can use os.path.

import os
parent_dir = os.path.dirname(os.path.abspath(__file__))

Not that bad, right? Well, this is just the directory of the file. Imagine you want to go a bit lower; you have to wrap that thing in another os.path.dirname invocation. Alternatively, you can use pathlib for a smoother approach.

from pathlib import Path
print(Path(__file__).resolve().parent)

Easier, right? But what does it have to do with dunders anyway? Well, pathlib also introduces one neat trick to help you create paths. Let’s say you want to create a path from your working directory to the directory foo that is in the directory bar. In pathlib you can do it like this:

p = Path('/etc')
p / 'init.d' / 'redis-server'

Dividing the Path object by a string will work similarly to our Appender class. It will join path and return a new Path object, ready to be joined again with another string. Thanks to the usage of the truediv (or div if you are still using Python 2.x) we can create paths in a way that is intuitive and similar to how paths are represented in string format. Here is how it looks in the source code:

def __truediv__(self, key):
    try:
        return self._make_child((key,))
    except TypeError:
        return NotImplemented

Interested in working for our software development company in Poland?
Check out our current job offers now

ORMs love dunders

If you’ve ever used any Python web framework, you’ve probably also used ORM for easier database handling. During my work at Profil Software - Python Development Company I usually use Django as my backend web framework and with it - a built-in Django ORM. If you were using other, less feature-complete frameworks (like Flask or Pyramid) you would probably use SQLAlchemy. Both of these libraries heavily use Python magic methods to make database usage as painless as possible.

For example, creating a connection between a model field and an actual database uses something called descriptors. The get, set, and delete methods are, respectively, three of the dunders that control what happens when values are read, set or deleted. Overriding each of these magic methods controls what happens when you manipulate model fields.

So, when you have a User object and you try to access a value of the related field, the code in the get descriptor is fired, data is fetched from the database, and you get straight to the value. Behavior like this is possible because of descriptors. The entire descriptor function for Django models is quite long so I won’t place it here, but you can see it directly on GitHub.

This is only one example. ORMs use these mechanics heavily. Let’s see how you would create a query using SQLAlchemy using a NOT statement. You actually have two options of doing that. If we want to get users that don’t start with “bob”, you can do this:

from sqlalchemy.sql.expression import not_


User.query.filter(not_(User.username.startswith('bob')))

Alternatively, you can skip the import of the NOT expression and use a bitwise NOT like so:

User.query.filter(~User.username.startswith('bob'))

Both will produce the same outcome thanks to ColumnElement subclasses implementing their own invert dunder that controls operator behavior. With this you can skip the import of the not_ expression and reduce the length of your code line while still preserving the general explicitness of what you want to achieve.

With great power comes great responsibility

This is, of course, just a small sample of what can be done with dunder mechanisms. If you know of other interesting implementations of operator overriding, let me know in the comments section, and if you want to read more about Python and coding check out our articles on 2 Questions to Ask Before Choosing Python Frameworks and 10 things you need to know to effectively use Django Rest Framework.
Python is a language that gives you a lot of freedom in what you do with your code. Dunders allow you to get extremely creative with how you approach various problems and let you to write very intuitive frameworks, ORMs and libraries for you and others to use. It lets you write code that looks natural even for someone that doesn’t have a lot of experience in programming. It also allows you to write an ugly, obfuscated code if you want to be a bad person.

Dunders are a feature that allow you to do both of these things at the same time and that’s why it is so important to keep in mind that readability, intelligibility and widely accepted good practices should be paramount in your code.

Profil Software is not only a Python Development Company, we also offer React Native Agency and DevOps Services, and we cover these subjects in our blog as well so feel free to check it out.