How I designed an offline-first app. An outline.

Last week I wrote a blog post on how I built my app in React, React Native and NextJS. This blog post gives an insight on how I made it into an offline-first app. There are many ways to build an offline-first app so this is a general outline of how I built it and what worked for me. I use NoSQL database so I use the word 'documents' throughout the blog post, but you can think of them as a record of a table in a SQL database.

Offline-first? What does that even mean?

I had to understand what offline-first apps really meant. I found different definitions of it around the internet. Partial offline functionality, partial offline data etc. but I wasn't satisfied with any of those, so I settled with the following definition:

Offline-first apps are apps which can run and function completely offline or without needing the internet for an indefinite amount of time. To offline-first apps, providing all functionality offline is the primary objective and any online functionality such as syncing to cloud is secondary.

There's also another category - offline-tolerant. Offline-tolerant apps provide functionality offline for a limited amount of time or provide partial functionality and sooner or later they would require the user to sync data to the cloud. The amount of time is dependent on the type of functionality of the app and how the data is stored. Offline-tolerant apps mostly store partial data in a temporary cache, whereas offline-first apps store all it's data in a dedicated local database.

Keep. Things. Simple.

Offline-first architecture can get overwhelming, so I made sure to keep things as simple or primitive as possible when I started out. I didn't get into conflict resolution strategies or tried to handle poor network connection immediately. I worried about that stuff later.

I worked with happy path and assumed that there were only two things I need to take care of - online and offline. When the app is offline, I track actions performed by the user. When the app is online - I replay those actions.

This might seem a bit different compared to conventional way of doing things which is to track "changes" instead of actions. Tracking actions was so much easier than tracking changes. I don't have to keep a record of hundreds of changes a user might have made to a document in the database. I only track actions and replay them. That's it.

Let's look at the following scenarios

When the app is online

• User performs an action (add, modify, delete etc.).
• Store changes in local database.
• Push changes to the server.

This is straightforward. When the app is online, I just push out changes to both local database and server.

When the app is offline

• User performs an action.
• Store changes in local database.
• Track actions in a queue and also store them in the local database.

When the app is offline, I track what action(add, modify, delete etc.) was performed and the unique Id of the document so I can retrieve it later from the local database.

When the app is back online again

• Get tracked actions.
• Replay those actions one by one skipping local database and push them out to server.
• Retrieve data from the server and merge the data.

I get the actions either from the local database or from the queue if still in memory and call the functions corresponding to those actions one by one. Each of those functions now also know to skip the local database and to call the server API directly. Finally, I retrieve the data from the server and merge it back into the local database (more on this later).

It all seems doable right? Keeping things simple was key here.

How should I detect if something changed?

I needed to figure out how to track what documents changed. I tried following techniques:

• Storing timestamps when the document changed and then comparing timestamps.

  I didn't go with this one because there were lot issues with this technique. What if a document was changed at the same time from two different devices. It could happen when there are multiple users modifying data or if the date and time of the devices are out of sync(its rare but it can happen).

• Versioning documents.

  Every time a change is made, a new version is created and the latest document along with version history is pushed out. I didn't go with this either as this would've made things too complicated, again I wanted to keep things simple. Git and PouchDB/CouchDB do this and they both do it in a really efficient manner, but I was using Firebase not CouchDB for reasons which are out of scope for this blog post. I needed a new strategy.

• Generating a new changeset ID each time a document is changed.

  Changeset ID is just an ID which changes whenever anything changes in that document. If changeset ID is different, that means something has changed so the document should be updated. This technique was simple enough for me to experiment with and implement so I went ahead with this approach.

Conflict resolution strategy

Now, I needed a strategy to handle conflicts. There were two I could think of - either I merge all the changes coming in, or I take last write wins(LRW). I went ahead with last write wins. The strategy you pick is dependent on the type and importance of data you're merging. If you are building a note taking app then merging text data would make sense.

In my case, I was developing a personal Kanban app and only a single user would be syncing data to other devices. Last write wins made sense in this situation. If something got overwritten, its expected that the user knowingly made the change and would fix the changes if necessary. Its far easier to deal with LRW strategy when syncing data both ways. Keeping things simple.

Syncing and merging documents from the cloud

With everything I now had, i.e. unique reference Id for each document, changeset Id to detect a change in the document and LRW strategy, syncing documents with the local database became straightforward. Since I was using Firestore, Firestore query listeners gets called when something changes on the cloud. Think of them as an event listener which are called when Firestore SDK detects a change. If I wasn't using Firestore, I would build some kind of polling mechanism to detect any changes on the server side.

To sync data, I do two things - Push first, then pull. Push the pending actions in queue to the cloud if there are any, then pull the data from the server. Pushing and then pulling makes things simple as this way the user's data is always up-to-date. The recent changes made by the user don't get overwritten by the changes on the server. This also aligns with my LRW conflict resolution strategy.

Pushing data to the server

I've already talked about pushing the actions before. You just call the corresponding server API functions and push the changes while skipping local database.

Pulling data from the server

To pull the data I employed two methods here:

• Getting all the user's documents from the cloud and comparing them with local database to identify which one got added, modified and deleted, and then updating the local database accordingly.

  This is a very broad technique, I made it more efficient by limiting the number of documents I get based on a subset of data, you'd have to figure out based on your needs how you can limit the amount of data. In my case, I was working with Firestore query listeners, each collection would have different query listeners and I wanted to work with minimum amount of listeners as possible so this technique works for me. I use this technique for my desktop app as I want "all user's data" to stay up-to-date.

• Only getting added, modified and deleted documents for a collection/table.

  This strategy worked when getting all of the user data wasn't necessary. Especially in mobile app, to conserve user's bandwidth, the app would only retrieve data which the user wanted instead of fetching everything.

Merging documents

Merging documents from the cloud to the local database involves adding new documents, updating modified documents or deleting "deleted" documents. Remember, I had unique reference ids and changeset Ids for each document? I would iterate through the both local data and retrieved data(from the cloud) and compare the changeset Ids, and then update the corresponding document in the local database if need be. It was time consuming to write the logic but it wasn't that bad.

Here's what I did for each case:

• Detecting new documents: If a new document is on the cloud, iterate through local collection, check if reference id exists, if it doesn't, its probably a new document so add it to the local database.

• Detecting modified documents: Compare the changeset Ids, if changeset Id is different, update the document in the database.

• Deleting "deleted" documents: By "deleted" documents I mean documents which don't exist on the cloud anymore. To delete those documents, for each local document iterate through cloud's data and find out if it doesn't exist, then delete it in the local database.

That's all for now

That's it for an outline. Using changeset Ids to detect changes made my life a lot easier. I also use them in the mobile app for comparing and updating data on the global state which improved overall performance of the app. There are so many things I didn't mention here as it would make the post too long. Besides if you don't do some research on your own, you won't learn ;)

Follow me on Twitter for behind the scenes of the app I'm building. I love connecting with new people and learning something new.

All the best!