A few sneak peeks into Hey.com technology (VII - Template page updates continued)

Update as of Dec 2020: now that the Hey support code has been extracted and officially released under the name Hotwire, we know that this article actually speaks about the Turbo Streams. Basecamp renamed and polished a few things but the principles stay the same as described here.

Remember the “puzzle” from the last section of part VI? Let’s finish it first.

The “Mark seen” response processing (resolution)

So, we’ve seen that once we mark an email as seen, Turbolinks asks the server for an updated HTML of it and the server tells Turbolinks to put it in front of all emails in the Imbox list. How come we see the mail in the middle of the list then?

If the server does not position the email precisely then something else must be used to reorder the listing. As you probably already guessed, it’s client-side JavaScript code that reorders the emails. Note that the page updater library does not even have any option to insert something in the middle of a list of elements.

And actually, somehow it makes sense that it’s not purely server’s responsibility, at least in this use case. To specify where exactly to insert a given email, the server would need to precisely know the seen / unseen state of the whole list on the client(s) which is very hard to do when we have possibly multiple parallel asynchronous actions going on, each affecting the emails list in a fundamentally unpredictable order.
A more brutal option would be to replace the whole list but that would be a step back in the direction of full page reloads with severely less interactive feel!

The reorder process itself is quite nice and straightforward: the whole emails list is guarded by a generic Stimulus controller called sorted_controller. It uses the Mutation observer to watch for additions and removals of the children elements in the list. Upon each mutation, it reorders the elements in the list according to the data-sort-attribute that it expects in each of them.

When we compare the original unseen mail element with the updated one returned from the server, we will notice that, indeed, they differ in just two little details: the returned email has the data-seen attribute set and a different data-sort-code:

As we observed earlier, the data-seen attribute serves for “splitting” the emails list into two separate ones (the “New” and the “Seen” emails). The sorting attribute handles the rest and effectively encodes a double ordering: the first digit seems to be either 1 or 2 based on whether the email is new seen or unseen. The rest of the number is just an updated_at timestamp.

Let’s stop here for a moment as this approach tells us something about Hey team’s convention for writing Stimulus controllers. At first glance it seems strange to encode the seen / unseen information twice in the email elements (in the data-seen as well as data-sort-code attributes). It seems redundant, not DRY enough… Let’s discuss this briefly:

• Couldn’t they use just the timestamp in the data sort code and tell the re-ordering controller to use both attributes? Well, of course they could but either the controller would have to sort by the seen attribute explicitly (and lose its generic nature) or they would have to pass it all data attributes to sort by in the HTML. This would work but would further clutter the HTML and would make the Stimulus controller a bit more complex, too.
• Instead, the Hey team chose the famous Rails approach of (a presumably simple) convention over configuration and they made a sorting controller that deals with just a single data attribute and doesn’t care about anything else. Then, if you need to sort in two dimensions, you just encode both of them in the sorting attribute. That’s it!

We’ll finish this section about the Mark seen feature with a funny little oddity: remember how the “PREVIOUSLY SEEN” header is shown entirely via a CSS rule that triggers upon the data-seen attribute in the mails listing? If we put a debugger breakpoint in the sortChildren action of the sorted_controller, we can stop execution right after adding the updated email but before the list is re-sorted and what we see then is quite funny, I think 😉:

Template page updates vs. Turbolinks frames

Warning: some free-form speculations here from now on… We must certainly wait for the official voices to illuminate the rest of us!

I really wonder why there are two very similar features in Hey to do partial page updates - the “turbolinks frames” (see parts III and IV) and the “template elements”?

Both of them are independent of Turbolinks. I mean both elements cooperate with Turbolinks but, in essence, the "frames" are custom HTML elements (which will work in any modern browser) while the "templates" are backed by a separate small JS library.

Both support replacing content on the page via a response from the server. The "templates" are, however, more universal, as they support not only replacement but a few other types of amending the page DOM.

Still, both require you to write no custom JavaScript at all to do the partial updates − all is governed by standard HTML stuff, links, forms and automatically handled AJAX requests.

Overall, to me the "turbolinks frames" feel a bit less generic, the associated JS code includes explicit functions to autoscroll to the element, enable / disable it, mark as "busy", etc…

The only distinct feature that I find missing in the "template" elements is the auto-loading of frames just after page load.

So, either there are some nuances that I didn’t get while digging through the source and that make the existence of both elements well-founded.

Or, and I would bet this second option is more probable, it’s just an evolution thing and the "template" will supersede the "frames" eventually in the official releases of Rails or Turbolinks. Besides the above-mentioned auto-loading, I can’t think of a technical reason that would impede rewriting a turbolinks frame into a template element. Well, we’ll see!

Fragmentation − a possible downside of the “composition pattern”?

Let me finish today with a tiny little rant: I really like the “composition pattern” that we talked about in the previous part: a more complex interactive behavior on a page is composed of multiple, very small, well defined, generic “bits of behaviors”, most of which are configured and put together in the HTML text of the page. It’s like the composition (over inheritance) approach in ruby, or like the “Do one thing and do it well” principle in Unix which encourages you to routinely combine multiple small programs with a shell pipe to get some more complex processing result…

And today, we could see this nicely in the Mark seen feature in Hey: you put up a link on the page, then, upon clicking, the server responds with the <template> element(s) and proper content-type header, a separate Stimulus controller reorders the result and a CSS rule brings in some subheaders. And all this is tied together in pure HTML code! (Plus conventions…) HTML becomes the carrier not only of page content but also behavior, an area that was traditionally a sole domain of JavaScript.

Again, I think that all of this is good and will help building sustainable non-trivial websites. I just think it’s kind of… fragmented. You need to look into multiple places to understand the complete behavior plus you need to learn and understand the conventions that the various bits communicate with. In a way, this has always been true in web development (and Rails especially!). But, it seems to me, the Stimulus / Turbolinks / HTML element triad that is heavily used in Hey website, kind of pushes this principle a long way further. I can tell you it took me quite some time to decipher, untangle and make sense of all these relationships among the various bits of code! Is it possible to cover these features with a small set of clear and nicely explainable default conventions or preferred page building styles?

But I don’t want to sound pessimistic. Over time, I actually got used to this style of writing pages quite well and I feel like it already changed my mindset towards expecting compositions anywhere on a page, from tiny bits of structure and behavior. Also, I expect the Basecamp / Hey team to come up with some really nice back-end DSL for all this new partial page update goodness − e.g. auto-handling the new type of templates (the template elements) simply by their filename extension, nice link / form helpers for the turbolinks frames (if they get released), a seamless cooperation with the websockets channels, etc. Overall, I think we have stuff to look forward to!

BTW, I would love to hear your thoughts on these topics… Feel free to add some!