What I learned from building a web framework.

Several weeks ago, I have been working on a new project Yaat (Yet Another ASGI Toolkit). It is an open-source asynchronous ASGI web framework written in Python.

I have been working with several web frameworks for so long especially Python frameworks like Flask and Django. Although I work with those, I do not really know how they are working behind. The journey started as a way for me to learn the internals of a web framework.

Inspiration

My inspiration comes from Jahongir Rahmonov’s blog post about how to build a web framework. Also, it is the starting point of the journey.

Before I continue further, let me give you a brief about Yaat!

Introducing Yaat

Yaat is an ASGI web framework designed to build web services using asynchronous programming in a simple way. It supports Python 3.6+ and licensed under LGPL.

Design Philosophy

Yaat is aimed to be batteries included, beginner-friendly framework. Anyone with a basic understanding of Python should be able to understand the documentation and kick-start the project in a short amount of time.

In addition, Yaat supports the loose coding style. It does not unnecessarily force any solution to the developer, and how the project should be structured is decided by the developer, not by the framework.

Features

Yaat already has views (functional and class-based views), requests, responses, routing, WebSockets, static file serving, templates (Jinja2), middlewares, CORS support, ASGI lifespan events, background task runners, test client using httpx, also has OpenAPI generator and SwaggarUI.

You can check out the Project Board for progress.

Say “Hello World”

Hello world in Yaat is as simple as …

from yaat import Yaat
from yaat.responses import TextResponse

app = Yaat()

@app.route("/")
async def index(request):
     return TextResponse("Hello World")

Now you know what Yaat is, enough talking about it. The main purpose of this article is to share what I learned throughout this project.

The Journey Begins

To be honest, It didn’t even start as a proper project with detail planning in terms of design and features. I just read how to build a basic WSGI framework from Jahongir Rahmonov blog and decided to build an ASGI framework.

As I have mentioned on top, it started as a way for me to learn the internals of a web framework.

ASGI (Asynchronous Server Gateway Interface) is the successor of WSGI. It is a design specification for how the web server should communicate with asynchronous web applications.

This is also where web frameworks like Yaat come in. It is designed to provide an interface for the developer to write asynchronous web applications easily.

Reinventing the wheel

Some say “do not reinvent the wheel”. But in my opinion, it is just wrong. We didn’t get the rubber wheel in a first try, it started with stone and then it improved over time. How did it improve? because people reinvent the wheel.

And that’s where I was on Feb 28, pushing the first commit to my new GitHub repository that later became Yaat. On my first day, I went straight to ASGI documentation and started reading to get a general idea.

Even though the initial project structure was based on Alcazar (framework Jahongir Rahmonov built in his blog post), Yaat is heavily inspired by Starlette (created by Tom Christie, a core contributor of Django REST Framework and httpx).

Release to PyPI

On March 23, I released the very first version of Yaat to PyPI. People can simply install by pip install yaat and start building a minimal asynchronous web application.

Writing documentation

Documentation is important. It does not matter if it is a close source project or an open-source project. There is no way someone will read the source code just to know how to use the tool.

I created a README file with proper instruction on how to install the framework, what are the requirements and features. Then I used GitHub Wiki to host the documentation. However, as the project grows, I start to realize GitHub Wiki is not suitable. I do not find the sidebar very useful when you have a lot of content.

That is when I realized I needed proper documentation. So I start looking for a new service to host the documentation. There are 2 requirements

1. Markdown support (I love it, and can back up easily)
2. Flat-file documentation (I want a service that can just read the markdown files and present it nicely on the web page)

I found GitBook at first, even though it supports markdown, it does not meet my second requirement. The winner for this is ReadTheDocs, it met both requirements and with MkDocs + material theme, I managed to published better documentation than GitHub wiki.

Tests, Tests!

No one is perfect. You cannot say the code is working as you expected until you finish writing the test. It applies the same to Yaat. As it is Python project, obviously I used PyTest.

If you look from the user perspective, who would want to use a tool that has no test. No one knows how reliable it is. Writing a test not only discover bugs you didn’t see at first but also it improves the quality of your code and gives confident to your users.

Also, measure the test coverage. I used pytest-cov and CodeCov.io to see the coverage reports.

Code Quality

Your code should be clean, readable. It is not something that runs normally but when comes to enhancement or changes, it is very painful for the developer.

(original post from reddit)

I use black (code formatter) and flake8 (code checker) with a pre-commit hook. So that it automatically format my code and flake8 checks my compliance to PEP8 (Python Style Guide).

Create PR (Pull Request)

Even if you are the only contributor to the project, always create a pull request. It is not a good practice to push everything to master. In my point of view, the master branch should be for code that is well-tested and stable enough for production. Believe me, creating a PR will save your time when you have to trace back a bug and will keep your product stable.

Whenever I have to implement a new feature, I branch out from master and always give names by feature-<what-i-am-implementing>. When I create PR, I give a short description and assign a tag to it. For example, enhancement for small changes to existing feature.

Automation

Running pytest on my local is great however I need to make sure Yaat is working as expected on different Python versions. My current machine is running with Python 3.8 so when I run the test I know it works on the latest Python version. But how about Python 3.6 and 3.7?

That is where Travis comes in. I use Travis to automatically run the tests I wrote against 3 Python versions (3.6, 3.7 and 3.8) so that I can confidently say that Yaat support Python 3.6+. And also generate a test coverage report and send it to CodeCov.io.

Thanks to Travis, I found a few bugs that occur only on Python 3.6. For example, asyncio.create_task is only available on Python 3.7 and above and for Python 3.6 you have to get the event loop first and create a task from the event loop instead.

Conclusion

I learned a lot from working with this project. From web internals, ASGI to tools such as pre-commit hooks, CodeCov.io code coverage. However, the chapter about the Yaat is not over yet and there are more features on the way to be released in the near future.

Feel free to check out the Yaat repository and give me feedback for any improvements. Thanks for reading this article!