The idea for django-unicorn all started innocently enough with a tweet on July 8, 2020.
After jealously watching demos of Phoenix's LiveView, I built a prototype of a real-time monitoring dashboard for Django with websockets and Alpine.js. After a past side-project got a little derailed (read: became not fun) using Django, Vue.js, and GraphQL, the simplicity of Alpine's model struck a nice middle-ground.
Then, I noticed the author's Livewire project. Even though it was for the PHP web framework Laravel it sounded intriguing and I was immediately smitten with the documentation site. The thought of simplifying web development by enabling server-side code to be "called" from the front-end was appealing. Instead of building a suite of APIs, mapping data models to their REST representation, and switching languages to build a single-page app in Javascript, Livewire
leverages backend code and provides the glue for the frontend to interact with. This fulfilled a need I saw all over the place -- it certainly isn't ideal for every application, but probably useful for 80% of the websites out there.
After watching the available screencasts I really wanted to at least prototype a project with it. But... not enough to switch away from my typical tech stack of Python and Django. Laravel
looks nice, but I am pretty invested in the Python ecosystem. So, I tweeted complaining that Django didn't have a similar library and my friend, Michele, then replied with the magic question: "why don't you make it yourself".
I spent the next 3 days re-watching the Livewire
screencasts pretty intently to see the "shape" of the JSON request and response messages, scouring the documentation site, and reading through the Livewire Javascript code to understand how it worked. My first push to Github was July 11th -- three days after that first tweet.
I remember how magical it felt to type into a textbox and then have Django render it in almost realtime as a regular Django template variable. Since Django's unofficial mascot is a pony, django-unicorn
seemed like a fitting enough name for this little library I was starting to mildly obsess about.
There have been a lot of learnings over the last five months. I'll cover a few related to Python, Javascript, and then some general thoughts now that django-unicorn
has grown up a little bit (version 0.11.0 was just released).
Python
Python has been my preferred programming language for the past 8 years or so, and Django has been my steady go-to web framework. There might be some flashier web frameworks around, but for the raw speed I can go from idea to database tables to server-rendered HTML I wouldn't pick anything else.
importlib
importlib.import_module is the mechanism to dynamically import Python modules. django-unicorn
uses this functionality to be able to find and load the component based on the component name's string representation that is specified in the template.
{% unicorn 'hello-world' %}
The hello-world
component name is converted to a module name of hello_world.py
and class name of HelloWorldView
. Then different Django apps are searched to find the correct module (defaults to unicorn.components.hello_world.py
). Once the whole string is created, import_module
is called to retrieve the correct component.
inspect
Python contains a wealth of information about the code that is running... if you know where to look. The inspect module provides a wealth of information about classes and its methods which I use to inspect for publicly available methods and fields to include in the Django template context.
literal_eval
django-unicorn
supports calling methods from the frontend with Python objects as arguments.
<div u:model="dictionary">
dictionary.name: {{ dictionary.name }}<br />
<button u:click='set_dictionary({"name": 1, "nested": {"name": 2}})'>set dictionary</button>
</div>
The method arguments look like Python, but are actually strings because all interactions are via JSON. The argument in set_dictionary({"name": 1, "nested": {"name": 2}})
needs to be parsed. Originally, I built a simple parser to convert strings into Python objects, but then stumbled onto literal_eval which "can be used for safely evaluating strings containing Python values from untrusted sources without the need to parse the values oneself." It "may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None", but I end up manually handling datetime
and UUID
as well. Much safer than calling eval()
and more sane than trying to handle all the cases yourself.
lru_cache
The standard library provides the lru_cache decorator that saves the results from up to the maxsize function calls. Once the maxsize+1 unique function argument is called, the cache evicts the first object that got pushed into it. cachetools provides similar functionality as a class so it can be used without the function decorator.
LRU caches are used in django-unicorn
to prevent re-finding and re-constructing component classes, and to prevent re-serializing the same data from a Python dictionary to a string representation. Both processes can be comparatively slow and tend to happen multiple times with the same inputs.
typing
Typing is relatively new to the Python ecosystem (introduced with PEP 484), but I find them to be a useful addition, especially with mypy and an editor that understands the type annotations (personally, I have been pretty happy with VS Code
and pylance).
Optional type annotations are only designed to help developers understand the code (they aren't used by the runtime for optimization -- at least not yet), but even so they have been useful for "future me" to better understand the context for my code. Coming from a previous static language, C#, I appreciate types to a certain degree, but I find this middle-ground to be particularly useful -- I have the freedom to prototype without a rigid type system in place, but as the design solidifies I tend to add in appropriate types where they might be useful.
Other helpful Python third-party packages
-
orjson: chosen because 1) it appears to be one of the fastest JSON serializers for Python, and 2) it provides library support for serializing more datatypes than the out-of-the-box
json
library (plus, it provides a hook to "dump" other types as needed) - shortuuid: used to create a unique identifier for components. The potential for possible collision is acceptable because of the limited potential number of components
-
wrapt: decorators are easy to create, but deceptively hard to make correct and
wrapt
handles all the hard parts - beautifulsoup4: sanely parse HTML without tearing your hair out
Javascript
I have primarily been a backend developer for most of my career and, besides a few side projects written in the early years of Node.js, I haven't worked substantially in Javascript besides adding little features here or there. Yet, as Michael Abrahamsen writes in his post about Flask-Meld, "...here I am, writing a whole lot of JavaScript so that I can write less JavaScript. I am an engineer, after all." It's a funny thought and I wouldn't say that I'm a particularly great Javascript developer, but I have learned a ton over the past 5 months about the DOM and the more "modern" Javascript ecosystem.
ES6
I'm not ashamed to say it: for a long time I didn't "get" the reasons to use anything other than ES5 Javascript on the browser for a long time. I didn't understand why I needed classes when I could do the limited DOM interactions I needed with prototype inheritance and functions. I also chafed at what I assumed was the requirement to use a transpiler like babel
for what seemed like such minimal benefits.
In fact, when I first started django-unicorn
it was all written in ES5 with lots and lots of unwieldy functions everywhere. Over time, it became really hard to follow the flow of code between all of the functions and I coudn't organize the code into understandable parts. Maybe it's just the "object oriented" mindset that has been drilled into me over the years, but I found Javascript modules
and the ability to use class
to be extremely useful to organize the code.
babel
and rollup
Since the modern browser support is almost universal for the ES6 features I use (95%+) I can develop using ES6 Javascript and only transpile to ES5 when generating the minified version that I ship in the library. Originally, I just fought with babel, but quickly afterwards I looked for something easy to configure that could also minify the separate Javascript files into one file.
After looking at Livewire
code again, I realized they use rollup which looked like it would fit the bill. There was quite a bit of fiddling and reading about IIFE to understand what I wanted to do, but now the build process is quick and painless.
ESBuild
(potentially)
I also investigated esbuild because of the promise of even quicker build times. There is a ESBuild PR that seems to work as expected. It even creates a slightly smaller file size than rollup
. However, there doesn't seem to be a way to integrate babel
into the process and I am not quite ready to give up on ES5 for users on really old browsers. At some point, that trade-off will probably shift I expect, though.
ava
and jsdom
I wanted a Javascript unit test framework that was fast and low ceremony (I think I see a recurring pattern) and ava seemed the best option. ava
has been working great so far and fits my approach well.
One issue with testing Javascript is abstracting away the DOM so you don't end up needing functional tests like selenium
(although I spent some time with web-test-runner and playwright and they were impressively fast to spin up). However, jsdom allows my tests to have enough of a DOM to test interactions like click events without requiring an actual web browser running. I did have to add hooks so that certain parts of my Javascript could use the jsdom
DOM instead of the browser's DOM, but after that was added it seems to work well.
morphdom
Part of the magic of django-unicorn
is how the DOM gets updated. That only works reasonably well because of the work of morphdom. A super impressive library and also a core part of Livewire
, as well.
In general
Creating an open-source library isn't all about the code (as much as I really wish it was). I learned a few things that weren't related to either Python or Javascript, but about the entire process.
Start small
django-unicorn
started as a germ of an idea, "Livewire for Django", but that was a daunting task. However, I knew what I considered the core functionality the library should have and could "see" a way to make it happen from the beginning:
- custom Django template tag that finds a component class and instantiates it
- expose all public fields on a component class to the Django template context
- Django view method that accepts a JSON object with a defined API
- Javascript that listens to events, converts them into a JSON object, and calls the Django view endpoint
- Django pieces to wrap everything together into an app that could be installed
The overall goal was overwhelming, but my first commit was relatively simple. I started with the basic functionality and iterated to add more and more functionality over time.
Breaking up a large project into smaller, achievable pieces is the best (or maybe only?) way I know of to build daunting, complicated software.
Everything needs marketing
As much as open-source is lauded as some idealized meritocracy, it really isn't. Building something cool and just waiting for people to find it is an exercise in frustration. I think Caleb Porzio does a great job of this with Livewire
. Between "working in public" on Twitter, conference talks, interviews, and podcasts, it's obvious he understands how important marketing is for his numerous projects.
The whole Livewire
website is also marketing. The first page is basically a landing page "selling" the library and why you should use it. The API documentation is clear and concise and the coup de grâce is the screencasts. It's clear that he understands that different people learn in different ways: some want detailed documentation, some want tutorials, some want a visual of how the library works. The screencasts also subtly counter some of the developer push-back about this approach. It's all brilliant marketing.
I knew for django-unicorn
to be even moderately succesful it would need more than a GitHub readme with a GIF. Pretty early on I created a stand-alone documentation site with an initial landing page and comprehensive documentation with example code. It is also important to have actual components that developers can interact with and see just how well they work. I'm definitely not a designer and would love help to make the documentation site better, but having a standalone site seems key to encouraging more users to try django-unicorn
.
Just showing up
For better or worse writing code is my day job and my hobby. django-unicorn
incrementally gets better over time because of the time I spend on it. Some weeks that might be very limited, but the average is probably 10 hours a week. Each week I slowly add new features, improve unit tests, tweak site copy, respond to GitHub issues, and improve the documentation. It only works because I enjoy all parts of the process. Like most hobbies, showing up and slowly improving a project is how to build something great.
It also helps to have friends who nudge you to create a fix for a problem instead of just mindlessly complaining about it on Twitter!
Thanks for reading this far and I hope some of my lessons were useful on your journey! If you are interested in a full-stack framework for Django, please check out https://www.django-unicorn.com and consider sponsoring me on GitHub. 🦄
Cover image from photo by De'Andre Bush
Top comments (5)
is this working? says web server is down
Which link are you referring to?
django-unicorn.com/
Thanks, I just added a redirect to handle that.
Great job! Thanks Adam for introducing livewire into django. I look forward to seeing your work flourish and be able to participate.