Python, type hints, and future annotations

Not long ago, I came across a bug in one of my projects that highlighted very interesting changes in how Python handles type hints at runtime.

This made me dig into the future of type annotations, and what it means for libraries manipulating type hints at runtime. Spoiler alert: It has to do with PEP 563.

• The bug
• A brief history of type hints
  • Basics of type hints
• Introducing PEP 563, or what lies behind future annotations
• The explanation
• How to get the return type of a function after PEP 563?
  • In Python 3.10 and newer
  • In Python 3.9 and older
• A word of caution
• Conclusion

_{TOC generated with bitdowntoc.}

The bug

I have a Django API (django-rest-framework - drf) which uses drf-yasg to generate an OpenAPI schema at runtime. You don't need to know the intricacies of those libraries, just that API endpoints return serializers, and that drf-yasg inspects them at runtime to generate a schema with the proper fields and types.

Now, the problem arose with a snippet similar to this:

from rest_framework import serializers

class MySerializer(serializers.Serializer):
   # This allows to return something that is not a direct
   # attribute of a model, but a "computed" value (function)
   is_foo = serializers.SerializerMethodField()

   # The type hint (-> bool) is used by drf-yasg to
   # guess the proper type is "is_foo"
   def get_is_foo(self, obj) -> bool:
      return obj.is_foo

Here, drf-yasg determines the type of is_foo based on the type hints of get_is_foo. The annotation says bool, so this converts correctly to the OpenAPI schema {"is_foo": "BOOLEAN"}.

However, upon refactoring, I added the following at the top of the file:

from __future__ import annotations

Suddenly, the schema became {"is_foo": "STRING"}. What?? (Note that it took me a while to pinpoint that the problem came from this import, but let's keep the story short).

If you understand why, you can stop here. If, like me, you find this utterly confusing and want to understand what is going on, keep reading!

A brief history of type hints

(I digress briefly, so feel free to skip this section).

In 2006, Python 3 introduced a syntax for adding arbitrary metadata annotations to Python functions (PEP 3107: Function Annotations). (Ab)used by many 3rd parties in different ways, Python 3.5 finally formalized the semantics in PEP 484: Type Hints (Python 3.5, October 2014 - see also PEP 483 – The Theory of Type Hints) and added the typing module to the standard library.

Over the years, the type hinting system evolved, offering new syntaxes and meta-classes. This is not only drastically changing the way we write Python code, but also the way we ship it to the masses.

Since Python 3.7, packages can incorporate type information and stubs files, making the life of IDEs and type checkers such as MyPy, pytype, etc. easier (It does, however, make the life of library maintainer a bit more complicated). If you want to know more, read PEP 561: Distributing and Packaging Type Information.

This is to say, type hints are now really taking off. I would not be surprised to find type hints ubiquitous in a few years. Javascript is taking the same route, deviating from the purely scripted language to something more structured.

Basics of type hints

I wasn't aware of most of the intricacies of type hints until I tried migrating a large project to mypy (a good read if you are in a similar process), but I assume most of you are now familiar with the basics of type hints:

from typing import List, Dict, Optional

def foo(a: List[int], b: Dict[str, object], c: int) -> Optional[None]:
  ...

def bar() -> bool:
  ...

Note that foo can now be rewritten without imports, thanks to PEP 585 – Type Hinting Generics In Standard Collections (Python 3.9, March 2019) and PEP 604: Allow writing union types as X | Y (Python 3.10, May 2020):

def foo(a: list[int], b: dict[str, object], c: int) -> str | None:
  ...

Introducing PEP 563, or what lies behind future annotations

If you ever tried using new type hinting features in old Python versions or struggled with forward references, you may have stumbled upon (and used) the famous future import:

from __future__ import annotations

def foo(a: list[int], b: dict[str, object], c: int) -> str | None:
  ...

Adding this line at the top of the file makes the snippet "work" (that is, not throw exceptions) from Python 3.7. Ever wondered why?

Well, the only thing this import does is turn on PEP 563: Postponed Evaluation of Type Annotations:

This PEP proposes changing function annotations and variable annotations so that they are no longer evaluated at function definition time. Instead, they are preserved in __annotations__ in string form.

Treating all annotations as string at runtime have clear advantages, including:

• It removes issues with forward references. Annotations can now refer to a name that has not yet been defined.

• It gives a simple way to deal with cyclic references. We can import the names when running type checks (if typing.TYPE_CHECKING: <imports>) and leave them out at runtime.

• It improves runtime performances. Type hints are no longer executed at module import time, which is computationally expensive.

This little trick, turning type hints into strings, also explains why we can use new type hint features in old Python versions. Even if str | None is invalid in Python 3.9 (TypeError: unsupported operand type(s) for |: 'str' and 'NoneType'), the string 'str | None' is perfectly fine!

All those advantages explain why PEP 563 is planned to become the default behaviour soon:

from __future__ import annotations was previously scheduled to become mandatory in Python 3.10, but the Python Steering Council twice decided to delay the change (announcement for Python 3.10; announcement for Python 3.11). No final decision has been made yet.

The explanation

So, back to our bug.

For a long time, the inspect module was the best way to determine a method's return type using annotations. It is what the drf-yasg library uses in https://github.com/axnsan12/drf-yasg/blob/1.21.7/src/drf_yasg/inspectors/field.py#L616:

import inspect

def foo() -> bool:
  return True

sig = inspect.signature(foo)
assert sig.return_annotation == bool

This worked perfectly until PEP 563: Postponed Evaluation of Type Annotations came along. When enabled, all annotations are suddenly str, so that was previously the type bool becomes the string 'bool'!

How to get the return type of a function after PEP 563?

For now, library maintainers need to assume annotations may or may not be stringized, and support both. This has been made easier for Python 3.10, but the transition will take time. Let's review our options.

In Python 3.10 and newer

From the documentation:

Python 3.10 adds a new function to the standard library: inspect.get_annotations(). In Python versions 3.10 and newer, calling this function is the best practice for accessing the annotations dict of any object that supports annotations. This function can also “un-stringize” stringized annotations for you.

To turn on the "un-stringize" feature, simply pass the parameter eval_str=True :

from __future__ import annotations

def foo() -> bool:
  return True

import inspect
# eval_str does the magic
ann = inspect.get_annotations(foo, eval_str=True)
assert ann["return"] == bool

Under the hood, this new function looks up the foo.__annotations__ dictionary and calls eval on values of type str.

If you never heard of __annotations__, it is a mutable dictionary "mapping parameter names to the evaluated annotation expression". It was introduced in Python 3.0 in 2006 (see PEP 3107 – Function Annotations). As its name suggests, return is a special entry key reserved for return annotations.

Note that inspect.signature is also updated to support eval_str, but it just defers the work to get_annotations(), so better to call the latter directly.

In Python 3.9 and older

In older versions, getting the actual type is tricky, even before PEP 563.

First, as explained in Accessing The Annotations Dict Of An Object In Python 3.9 And Older, just getting the annotation has some flaws. Without going into the details, the safest is to use this code snippet:

def get_return_annotation(func):
  if isinstance(o, type):
    ann = o.__dict__.get('__annotations__', None)
  else:
    ann = getattr(o, '__annotations__', None)
  return (ann or {})["return"]

This can return a type, a str, or None.

In the case of a str, we may need to "un-stringize" it. Again, the doc gives some pointers in Manually Un-Stringizing Stringized Annotations, but the gist is to use eval and see if this works. But there are many catches... So what do we do?

From my own experience (and deviating from the docs), I suggest one of two approaches:

1. Either give a shot at get-annotations. This is a very small library that backports inspect.get_annotations() (with eval_str support 🤗) to older Python versions. It doesn't have many stars on GitHub but seems to deserve more.

• Or try typing.get_type_hints. Python 3.7's doc says "In addition, forward references encoded as string literals are handled by evaluating them in globals and locals namespaces", which roughly translates to "if str, I will try my best to still return a type".

A word of caution

future.annotations lets you use anything as annotations, including features that are not yet available (or even "garbage"). This is very dangerous if you plan on reading types at runtime.

For example, if you are using list[str] or str | None in a version that does not yet incorporate them, any attempt to unstringize the annotation will raise an exception (e.g. unsupported operands type(s) for |).

Thus, the safest (especially for libraries) is to stick with the features present in the lowest version of Python you support. In the example above, this means using typing.List and typing.Union instead before Python 3.9.

Conclusion

The advent of PEP 563: Postponed Evaluation of Type Annotations will greatly change the way Python code can access types as runtime, as it turns all annotations from type to str representing types.

Before this PEP becomes the default (Python 3.12?), it can already be activated by importing __future__.annotations. Any annotation in the same file automatically becomes stringized before being stored in the __annotations__ dict.

If a piece of code needs to get the return type of a function (or any other annotation) at runtime, it needs to take this new reality into account.

In summary, here are the different ways of properly getting types at runtime:

• inspect.get_annotations(f) → only available for Python 3.10+, it is the new best practice. It deals properly with stringized annotations when eval_str=True.

• inspect.signature(f) → properly get the types, but only supports eval_str since Python 3.10.

• f.__annotations__ → dangerous before Python 3.10, due to some quirks. Never access it directly unless you know what you are doing!

• get_annotations.get_annotations(f) → very small library that offers eval_str starting from Python 3.6.

• typing.get_type_hints(f) → states in the docs that it will do its best to deal with stringized annotations, but test thoroughly before use.

Keep in mind that stringized annotations let you write anything, but as soon as you try to unstringize them, exceptions will be thrown. So no str | None before python 3.10!