I am currently looking for a better way to save and retrieve and save data for my favorite side project. I did some research and I thought it might be nice to share what I found out so far.
Requirements
The app is meant to run as pwa as well as electron desktop app and uses rxjs to a wide degree. At the moment there is no backend and if there ever will be one it will probably be optional. As the main focus is the electron app I fortunately don't have to care too much about browser support, though all the libs seem to have support for all modern browsers.
These are the requirements I came up with:
Essential
- Super speed: it should feel as fast as a native desktop app for about 2000 different entities at minimum and up to 10mb of data stored and quick to load combined data and subsets of data
- Relatively easy to implement without too much overhead
- Future proof: should be under active development and supported by a big community
- No UI Blocking should occur even when working with larger datasets
- Framework agnostic
Bonus
- Working typescript support (at least no errors)
- Readable indexddb model => the data should be browseable via chrome dev tools
- Ideally should provide the ability to get a dataset as an observable or it should be easy to implement such thing
- Replication to a backend server: though not yet implemented, it might be nice to have the option
- Bonus bonus: simple file storage replication to google drive, owncloud or dropbox
The competitors
As a starting point I used this list. Please keep in mind that I am by no means an expert for any of the libraries used here, so chances are that I missed something important. If so please let me know in the comments.
These are my picks:
WatermelonDB
Description: Reactive & asynchronous database for powerful React and React Native apps
advantages
- Sync capabilities build in
- Quite fast thanks to LokiJS
- Includes a query language similar to mongodb
- Schema validation
- Database interactions run inside a worker so no risk of ui blocking
- Write sqllite to disk (which might be nice to use with electron)
drawbacks
- Typescript Errors
- Strongly focused on react
- Some issue with fast consecutive writes (but that might have been me using it wrong)
- IndexedDB is just stored as one big string
RxDB
Description: A realtime Database for JavaScript Applications
advantages
- Sync (thanks to PouchDB probably the most advanced)
- Nice Syntax
- Wide framework support and examples
- Schema validation via JSON Schema
drawbacks
- Very slow for larger datasets
- Render blocking occurs when interacting with larger datasets (this might be my fault again though)
- Data is stored separately but is not easy browsable because of all the PuchDB stuff
Dexie
Description: A Minimalistic Wrapper for IndexedDB
advantages
- Schema Validation
- Relatively fast
- Clean Database usage for IndexedDB
- Framework agnostic
drawbacks
- Not clear to me yet if syncing is easy to implement
- No real observable interface (yes there is dexie observable but, but that is just for watching change events rather then getting updated data)
LokiJS
Description: javascript embeddable / in-memory database
advantages
- Fast
- Includes a query language similar to mongodb
- Framework agnostic
- Less overhead than RxDB and WatermelonDB
drawbacks
- No sync capabilities built in
- ? No observable data interface ?
remotestorage
Description: An open protocol for per-user storage on the Web
advantages
- Schema validation
- Own your data: Sync to google drive / dropbox and more
drawbacks
- Out of date dependencies
- No active development in the last couple of months
- Smaller community than the rest
Native IndexedDB
advantages
- no overhead
- relatively fast
- framework agnostic
- typescript should work
drawbacks
- completely custom sync required
- No observable interface
- probably lots of code to write
- not all browsers support all the features
Speed Test
This might be outdated but apart from my own non systematic tests I used this tool to compare the performance.
Premature Conclusion
So far I've only experimented with WatermelonDB, RxDB and a little bit with native IndexedDB. Here are the branches for WatermelonDB and RxDB in case you want to check for yourself:
https://github.com/johannesjo/super-productivity/tree/feat/watermelonDB
https://github.com/johannesjo/super-productivity/tree/feat/rxdb
WatermelonDB put me off because of its strong focus on React and it not working well with typescript atm. It shares one of the "problems" with LokiJS: The data in stored in IndexedDB is just a big string. This might not be a real problem, but it feels wrong to go about persistence like that. I didn't come much further than playing around a little with LokiJS, because I felt that it might be basically WatermelonDB with less issues, but also less of the stuff I need and I was still hoping for something better.
RxDB looked pretty promising and I assume that it really shines when you are using a server. But the bad performance was just not an option for an app that aims at desktop app like performance.
Using native IndexedDB should work fine for the most part. The main disadvantage is that I would have to write a lot of code and I don't want to re-invent the wheel (just quite yet).
Next step is Dexie! The community might not be as big as the ones of it's competitors and the lack of a prebuild performant way to use collections as observables is a little bit disappointing. I am also unsure about the sync stuff, as I didn't find too many good examples and it seemed to as if it was more of a gimmick than something people are using in production, but I hope I am wrong! So far it has been the only library offering a clean IndexedDB to browse while at the same time offering some structure making sure I am not the only going about saving data this way.
I'll update the article once I know more!
Did I get something wrong? Probably! This why I'm very curious to hear your thoughts and experiences on the matter. Please share them in the comments!
Top comments (6)
How about GUN ?
Its a graph database, using only key-value's (so you need to rethink a lot of structures coming from a database / using lists all the time, been there too 😄)
But once your data works within a key-value-way, the real fun begins, gun is a decentralized database, so depending on a gun peer, you'll have data synced in real-time.
One of the biggest examples would be a reddit clone notabug.io/ this is fully created on gun and yeah amazing work :)
Thanks for the suggestion! I seemed to have a look at it some time ago (had it starred already on github), but I don't really recall why I dismissed it. Will have another look!
EDIT: I remember now. Gun seems aim at more dynamic use cases than a single user database. I asked about this on their gitter chat. If there is a good way it, I definitely give it a go.
EDIT2: Actually this looks super interesting and the community seems to be great. I'll keep you updated!
Hey, Great Post.
Have you heard about LowDB? github.com/typicode/lowdb
It is an JSON „powered“ database, which can use the filesystem or the LocalStorage.
I don’t know how Support vor Observable objects is because I didn’t dive deep into it when I used it in the past but it was the first that flew into my mind.
Please correct me if I missed something
I hope was was able to help you
Thanks for your reply! I had a quick look at it, but assumed that it would be slower than it's alternatives, because there also just seems to be a big string getting stored. Also there hasn't been much development going on in the last couple of months.
The case for LowDB looks to be "you're already bundling lodash into the frontend and need to do a little something stateful" which of course is not universally applicable. But that aside, disqualifying it on the grounds of a few months of inactivity isn't really reasonable -- some libraries are just mature, even in JavaScript.
You're right about that. It wasn't the main reason not use it. But I also would argue that it can be a sign that the maintainer has no or at least less interest in it. Other people might be better adjusted but I personally tend to 'improve' the software I care for even if I shouldn't :)