SQLite Database Locked - Live synchronisation of two databases with row-by-row UPDATEs/INSERTs

Question

We're currently experiencing issues with database locking using SQLite in a multi-user environment. These are coupled with regular database malformation issues, due to the fact that journalling has been switched off to reduce the number of database lock exceptions.

My colleague is currently implementing a synchronisation algorithm to allow users to work on a local SQLite database and have that data transferred to a master database on the network near-instantaneously.

When a local row is changed, a Sync flag is set to true. The synchronisation algorithm is triggered by a timer running every second, and consists of two methods: Upload and Download. Any SQL commands mentioned below are being built using String.Format.

During Upload, the local database is scanned for rows where the Sync flag is true. Each row like this is either updated or inserted into the master database (depending on whether a row can be found with the same PK).

During Download, every local table is compared, row-by-row, field-by-field, to the corresponding table in the master database (both tables are loaded into memory as array structures to speed up the comparison). If any differences are found, or the local row is missing, the local row is updated/inserted using the data from the master database

Ignoring the obvious fact that this will not resolve our locking/malformation problems, my intuition tells me that this is a terrible idea, but I'm having trouble explaining this to the-powers-that-be. Could anybody provide some reasoning as to why this would be a terrible idea, or perhaps some positives I've overlooked.

Answer 1

Switching off journaling does not reduce the number of locking conflicts, it just converts locking exceptions into guaranteed database corruption.

The correct way to hande locking conflicts is to wait until the other connection has finished making changes. When using System.Data.SQLite, this can be set as Default Timeout in the connection string and has a default of 30 s (I don't know what driver XPO uses). If that doesn't give you enough concurrency, you should switch to a client/server DB like MySQL or PostgreSQL.

The synchronization download is silly: you're reading the entire database anyway, so it would be easier and faster to just copy the entire database file to the local machine.

---

If you think that a word from The Authorities might have a bigger chances of convincing your PHBs, ask this question on the sqlite-users mailing list.