Concurrent writes with SQLite and Peewee

Question

I'm planning to use SQLite and Peewee (ORM) for a light duty internal web service (<20 requests per second). The web service can handle multiple simultaneous requests on multiple threads. During each request the database will be both read from and written to. This means I will need to have the ability for both concurrent reads AND writes. It doesn't matter to this application if the data changes between reads and writes.

The SQLite FAQ says that concurrent reads are permitted but concurrent writes from multiple threads require acquiring the file lock. My question is: Does Peewee take care of this locking for me or is there something I need to do in my code to make this possible?

The Peewee database object is shared between threads. I assume this means that the database connection is shared too.

I can't find a Peewee specific answer to this so I'm asking here.

Answer 1

Sqlite is the one doing the locking, although I can see how you might be confused -- the FAQ wording is a bit ambiguous:

When any process wants to write, it must lock the entire database file for the duration of its update. But that normally only takes a few milliseconds. Other processes just wait on the writer to finish then continue about their business. Other embedded SQL database engines typically only allow a single process to connect to the database at once.

So if you have two threads, each with their own connection, and one acquires the write lock, the other thread will have to wait for the lock to be released before it can start writing.

Looking at pysqlite, the default busy timeout looks to be 5 seconds, so the second thread should wait up to 5 seconds before raising an OperationalError.

Also, I'd suggest instantiating your SqliteDatabase with threadlocals=True. That will store a connection-per-thread.

Answer 2

Consider to run all writing operations within 1 async process. This made the Javascript server programming nowadays so famous (although this idea is know far longer). It just needs that you a bit familiar with asynchronous programming concept of callbacks:

For SQLITE:

• Async concept directly in Sqlite: https://www.sqlite.org/asyncvfs.html
• APSW (Another Sqlite Wrapper) which better supports SQlite extentions in Peewee: http://peewee.readthedocs.org/en/latest/peewee/playhouse.html#apsw

For ANY DB.

Consider to write your own thin async handler in python, as solved here e.g. SQLAlchemy + Requests Asynchronous Pattern I would recommend you the last approach, as this allows you more code portability, control, independance from the backend database engine and scalability.