Consolidating changes when syncing with a server from offline localStorage

Question

I am planning to use a localStorage adapter for backbone.us to allow it to sync/fetch to local storage instead of via jqXHR. This is so that my app can work offline.

However, once my app is back online, I'd make an ajax call to sync up the local dataset with the server, or alternatively build in some sort of a "replay" system to send only the changes over.

However, how would I handle when the data set has diverged (changed on both server and client)? Which source has the correct data set?

Answer 1

You should be able to create a simple two-way synchronization protocol by doing the following:

1. On all tables that should be synchronized, have a "last updated" timestamp field which is updated whenever you modify a row (including inserts and deletes, see further down).

2. For "id", avoid auto-incrementing integers (which would require talking to the server to figure out the next value), and use GUIDs instead. The localStorage adapter does this by default. Using a GUID, both the server and client can generate new rows without talking to eachother.

3. You need to use "soft deletes". Instead of actually deleting a row, have a flag that you mark as deleted and filter on this flag whenever you need to list your collection of objects. That way, even deletes gets propagated properly. You can do "housekeeping" (actually deleting the rows) for all rows that have remained deleted for a certain amount of time, where that time needs to be larger than the largest possible "offline" time to make sure deletes have propagated.

4. Store a local timestamp of the last time you sync'ed with the server on your client. Making sure you store timestamps on both the server and client with the proper timezones is important.

5. When you sync against the server from the client, send it your "last sync" timestamp, and the server should send you everything that changed since that time. With the new timestamp provided by the server, store that as your new "last sync time" locally on the client. Updated/insert received whatever rows you get depending on the GUID (update if you have them locally, insert if not).

6. If you have locally stored changes, send all locally modified rows since the original timestamp from your client to the server, preferrably before reading updates from the server (you can also do it after, but regardless, you should probably check the updated timestamp to figure out which update is more recent and handle it according to whatever conflict resolution policy you select, or none if it's not important).

7. Feel the pain when you realize that for practical cross platform localstorage usage, you are currently limited to about 2.5 megabytes of data. That will hopefully change soon. And before you decide to go IndexedDB instead to work around the limitations, the APIs haven' stabilized yet, and most implementations are half-assed and incompatible.

Answer 2

What about simply ensuring your data has some marker for when it was last edited? Then you can compare, and if the server is newer, overwrite the client, and vice versa.