The Offline-First Sync
Transferring data between devices is a common feature these days. Most apps you use everyday implement some kind of mechanism for syncing information between the app and the server it's connected to, enabling instant access, no matter where you connect from.
Syncing gets difficult the moment you make your app offline first. Your data, including all changes made on a different device, must be accessible offline.
One of our core values is to make your knowledge base available no matter the system you use. We've specifically tailored acreom to work primarily offline, with sync being just an extension for our offline-first system. No matter where you access from, whether using a web browser, mobile, or desktop apps, your data is primarily stored on your device and only then synced to your other devices. This ensures that you always own your data.
Offline-first sync comes with its own specific problems that pile above all the issues of other sync solutions. Engineers at Linear had an insightful talk about the problems and workings of a similar sync solution and so did Figma. We have decided to base our sync engine on a similar concept, with some interesting modifications.
Offline First
Before we dive into how the sync extension works, let’s start with covering the offline workings of acreom.
In most cases, offline-first apps store data either in a local database or directly on a disk. Internally, acreom uses indexedDB. Each entity modification creates a change object that is saved in the Changes Queue. A change describes what exactly happened to an entity. It contains the ID of an entity, the table it belongs to, the type of the change, and all the modifications. To make the sync work, three types of changes are used - create, update and delete.
{
key: <uuid>,
type: 2, // 1 - create, 2 - update, 3 - delete
table: “documents”,
modifications: {
…
}
}Change objects in the queue are thrown away the moment they are processed. The change queue allows multiple processors to consume changes. In our case, there are two: a Cloud processor that sends the change to the server, and a local file system processor that writes the data to the filesystem. Additionally, search indices are updated and any 3rd party apps are notified of a change if necessary. All of these are running as workers on separate threads.
Note that this process is invisible to the user. The frontend layer will update the data optimistically and instantly for a seamless experience.
Sync
The sync protocol is initialized when you log in. It establishes connection to our sync server, fetches all already synced data, and registers websocket listeners.
Syncing Between Devices
Websocket connection enables real time sync of data between different devices. acreom protocol communicates with the sync server in two ways:
The protocol takes changes on entities marked as syncable and sends them to the sync server using REST. It also fetches incremental changes on each app startup, ensuring the data present in the app is always up to date.
The second part of the communication is based on websockets. In case you have multiple devices online at the same time, changes are sent to devices using websockets, resulting in the devices getting the updates in real time.
All changes that are synced are grouped by the entity. If you edit a single entity multiple times in a row, these changes are grouped into a single change object.
Coming Online
The same approach is used when you do some work offline, and then (when online), try to sync. Each change gets saved separately and is resolved into a single change object right before being synced. Data is therefore synced almost instantly, no matter the changes you've made while being offline.
The Backend
The server side of the sync is quite simple: the sync server receives requests from acreom and compares the entity received to the one stored in the database. If the entity received is a newer version of the entity stored, it is saved and propagated to all online devices using websockets. If it’s not, the change is discarded. All devices that are not online will fetch the latest state from DB when they come online.
Conflict Resolution
The current conflict resolution of the sync is "latest version wins". We plan on bringing CRDTs and transactions to replace the existing implementation alongside the collaboration later this year.
Handling Network States
The biggest challenge we've faced was not the synchronization itself, but the management of the synchronization state. Disconnecting the protocol (e.g. when acreom goes offline, when there are connection issues on our backend, or when you lock your screen), turned out to be a bigger challenge than we thought it would be.
acreom is an electron app, so we rely on solutions available for the web. Electron window has a listener for online and offline events, but they are not reliable all the time. When you go offline but stay connected to the wifi, the listeners won’t fire and the protocol continues sending requests. This results in requests not going through and protocol getting stuck. Same goes for putting your pc to sleep. Proper handling of online/offline state became a priority.
While working on the mobile apps, we've noticed that iOS had a great offline detection. We've decided to look into it and use the same solution. The solution boils down to pinging different DNS servers in a few second intervals. The pinging is done after a request fails because of network error, or an offline event is received. After the first successful ping, the protocol tries to reconnect, but if it fails, the process is repeated from the first step Upon success, acreom protocol fetches all data that changed since last sync from the sync server, applies local changes on top of the newly fetched data, and syncs local changes to the sync server.
Takeaways
While building the sync protocol, we've obtained a few key insights, which helped us iterate to the right solution.
Managing sync state is key. Knowing when to sync and when not to sync will make your life much easier. Being able to rely on knowing when you are online and offline takes a big chunk out of the sync protocol complexity.
When syncing data in a "last change wins" kind of fashion, some scenarios are not worth solving. You won’t solve the problems that come with it completely and the minor gains are worth it only on paper, users won’t notice much of a difference.
Merging changes in text without CRDTs or transactions and without knowing the chronological order is not worth it. If you were to keep all the changes, just go with CRDT like Y.js and avoid implementing solutions from scratch.
It’s better to not sync one update than not sync anything at all. If you have issues on the backend with saving a certain entity, it's better to skip a single update and let the user know about it, than to stop the whole sync. Yes, you can fix the sync server, but the user can lose their unsynced data in the meantime, for example from changes that happened on other devices.
This blog is part of acreom dev week. Be sure to check it out and follow along. Check out also our twitter, or join our Discord community to stay in the loop.
