Staging SQLite Database Files Reliably

Question

I have two SQLite database files:

• data.db (Production)
• data.db.tmp (Staging)

Both databases are in WAL journaling mode. Additionally, the staging database in in exclusive locking mode (using PRAGMA locking_mode) with a single reader / writer, while the production database is in shared / normal locking mode and has several readers and no writers.

At any given point in time, the file structure might look like this:

• data.db
• data.db-shm
• data.db-wal
• data.db.tmp
• data.db.tmp-wal

Ocasionally, I will want to replace the production database with the staging database - preferably, without disrupting existing [production] readers and, more crucially, without corrupting the database.

My initial (naive) idea was a simple mv data.db-tmp data.db but, because there are several related files, a single rename will not guarantee atomicity nor consistency. I then thought of doing a braced mv:

mv data.db{.tmp,.tmp-wal} data{.db,.db-wal}

I don't know if the above is a atomic operation, but it wouldn't work as well given the transient nature of the *-shm and *-wal files: if data.db.tmp-wal doesn't exists the move would fail (I think!) and there is no atomic operation for the possibly existing data.db-shm counterpart.

According to info coreutils 'mv invocation':

Prior to version '4.0' of the fileutils, 'mv' could move only regular files between file systems. For example, now 'mv' can move an entire directory hierarchy including special device files from one partition to another. It first uses some of the same code that's used by 'cp -a' to copy the requested directories and files, then (assuming the copy succeeded) it removes the originals. If the copy fails, then the part that was copied to the destination partition is removed.

Renaming whole folders is also not atomic.

What can I do to make this staging process reliable?

---

Some additional remarks:

• my client API is PHP/PDO, so I have no access to the SQLite online backup C interface
• the databases are several GBs in size, so some in-memory solutions might not be viable

Answer 1

Your command expands to:

mv data.db.tmp data.db.tmp-wal data.db data.db-wal

Anyway, multiple file system operations are not atomic.

---

To get rid of both -wal and -shm files, change the journal mode to DELETE; you can change it back later. Once you have a single database file, you can execute an atomic rename to replace it.

Please note that WAL mode optimizes for writing; it would be a better idea to leave the (read-only) production database in rollback journal mode.