SignalDB: Bringing Meteor-Like Reactivity to the Modern Age

TL;DR

Check out SignalDB at https://signaldb.js.org.

It was in 2014 when I quit my job working with Microsoft SharePoint to transition to a more modern realm of development tools. I started working at a small agency, building project management software from scratch. A year later, a colleague and I spun off and founded TeamGrid. TeamGrid was built with Meteor.js and, with a few small exceptions, I was the only developer working on it.

Meteor

Working with Meteor felt like magic compared to the hacky jQuery-Injections in SharePoint I had done before. Initially, understanding asynchronous development with node.js was a bit challenging. Also, working in a Full-Stack environment was sometimes a bit confusing.

One aspect I was entirely impressed with was how easy it was to work with data. One reason definitely was that a subset of the relevant data was synced to the client, and on the client, it was possible to use the same API to query and modify the data, thanks to minimongo. Minimongo was a client-side implementation of a database that had a very similar API to MongoDB. You define collections and save documents to them. With selectors, you're able to query data in collections. Data was synchronized in real-time between the client and the server.

Working with Blaze for the UI was also very easy, thanks to the built-in reactivity with Tracker. Tracker is Meteor's reactivity implementation. Although a bit complex to understand when looking at the internals, it offered an outstanding developer experience when using it. Like I said, it felt like magic. Once the UI and the reactive function were defined, they would automatically rerun when the data changed, and the UI updated seamlessly.
Meteor's Optimistic UI also really impressed me. It anticipated the server's response, allowing the UI to update instantly, which in turn improved user experience by minimizing noticeable delay. It blended effortlessly with Meteor, enhancing its real-time data synchronization and reactivity capabilities.

One thing which really blew my mind, was the meteor-reactive-class package. With this package, it became feasible to define a class and convert a collection's documents to it, such that queries would yield instances of this class instead of plain objects. This facilitated the writing of class methods to fetch related objects. For instance, when iterating over posts, each document would be converted to an instance of the Post class. The Post class got a method named author() that retrieves the post's author based on the authorId stored in the post.
Interestingly, executing author() within a reactive context triggers a rerun of the context not only when any post data alters, but also when the author's data (name, email, etc.) changes. This significantly elevated the developer experience by streamlining data handling and ensuring seamless reactivity, which in turn, reduced the amount of boilerplate code and made the codebase more intuitive and maintainable.

New Frameworks

Time goes by, and in 2020, I left TeamGrid as well as the Meteor environment. A lot has evolved in the JavaScript world since then. Modern view frameworks have now become the standard. Nearly everyone writes JSX instead of separating the markup from the UI logic. The tooling for JavaScript apps has become much more mature and stable compared to the earlier days. I transitioned to React on the frontend as it was the most popular framework at the time I switched.

On the server side, I've attempted various solutions to fill the void left by the absence of Meteor. The community has been buzzing about GraphQL and Apollo. The concept of Apollo is appealing, particularly for retrieving data from a server. Especially the synergy of resolvers targeting different data sources is highly potent. However, while the architecture is appealing, the front-end implementation wasn't as intuitive and resembled the early days of fetching data from REST-APIs. There wasn't much tooling available to integrate GraphQL seamlessly into your frontend. Integration is mostly done at the component level, but ideally, you would want your data handling logic centralized, rather than scattered across various components. Although you can create an abstraction layer, bear in mind that you will need to maintain it. In this scenario, you're the sole individual generating innovative ideas on how to simplify and improve the process. It appears that most individuals utilizing Apollo and GraphQL have adopted this approach, which is acceptable.
Besides Apollo/GraphQL, I also explored frameworks like FeathersJS and tools like Firebase, Appwrite, or Supabase. Each solution had a robust backend implementation, but none matched the Developer Experience of Minimongo and Meteor on the frontend side.

About a year ago, I discovered a cool offline-first framework called RxDB. Initially, I thought that on the frontend side, this was exactly what I had been searching for over the past years. After tinkering around and even using it in production for some time, I realized that it wasn't well-suited for my intended use.
RxDB was initially created as an RxJS layer for PouchDB with a server replication interface. Over time, other storage types besides PouchDB were introduced (e.g., IndexedDB, SQLite) and the replication interface became more sophisticated. The replication interface is really cool and exactly what I wanted.
The biggest problem I have with RxDB is that it is so tightly coupled with RxJS. While RxJS is technically very powerful, it offers a dreadful developer experience. It's really hard to understand at first and integrating it into an existing codebase, which isn't using RxJS, is tedious.

Signals

A while ago, I encountered signals in SolidJS for the first time. I don't recall exactly when or where it occurred, but it was a few months after Ryan Carniato's "The Evolution of Signals in JavaScript" was published. When I came across the term "signals" in relation to reactivity, I instantly grasped its connection to the reactivity I had previously worked with in Meteor.
For those unfamiliar with signals, I highly recommend reading his other the article about signals. It thoroughly explains the concept.
Upon delving deeper, my initial understanding was confirmed. Further research revealed that many implementations of signals reactivity already exist within the JavaScript ecosystem.

SignalDB

Upon realizing that signal-based reactivity is gaining traction again, I began exploring how I could integrate it with a frontend database to replicate a developer experience similar to what minimongo offered during the meteor days. After some experimentation, I designed the initial concept for SignalDB. I aimed to employ a synchronous interface to maintain simplicity. Given this, it was essential to store all data in memory. Although this approach may not be ideal in terms of memory usage, it’s optimal in terms of performance. For the rest of the API, I took a lot of inspiration from mongodb and also minimongo. To apply the selectors of a query to the data, I discovered the remarkable library mingo, which replicates many of the mongodb selectors in JavaScript, making them executable on an array of objects.

The first functional version of SignalDB was completed within a few hours and was capable of creating collections, saving documents, and querying them.

Following that, the next step was to introduce reactivity into the system. While exploring various signal implementations, I noticed that most of them had some similarities, which made it possible to write a generic abstraction layer, called a reactivity adapter, with which any signal library can be plugged into SignalDB. I also started implementing reactivity adapters just for fun when I found a new library that implemented signals.
When developing reactivity for queries in SignalDB, I was highly inspired by the minimongo implementation. It is amazing how far ahead of their time the Meteor guys were.

Since all data were initially saved only in memory, I needed to consider persistence. To maintain flexibility in terms of persistence, I followed a similar approach as I did for reactivity by implementing an abstraction layer. This allowed for the integration of any type of persistence interface. Initially, I implemented a local storage adapter for the browser and a file system adapter for Node.js. I encountered a challenge with the replication functionality, as finding a modern and general approach is difficult. Then I realized that I could postpone the replication functionality to a later point and just implement a persistence layer that saves the data in RxDB. This turned out to be a wise decision as it simplified the implementation of SignalDB significantly.

I have recently started using SignalDB in production and gradually phasing out RxDB. I am quite satisfied with the results, and I believe it serves as an excellent alternative to RxDB. Moreover, I am highly pleased with the developer experience, as it is remarkably straightforward to comprehend and integrate into an existing codebase. The one remaining obstacle preventing me from fully transitioning away from RxDB is the replication interface. Currently, I utilize RxDB as the persistent storage for SignalDB, which simplifies the transition process, requiring me only to replace data queries and incorporate signals instead of rxjs. For implementing signals, I rely on maverick-js signals and have developed a React hook that abstracts the interface, making it convenient for future transitions.

Learn more about SignalDB and also check out the documentation at https://signaldb.js.org