zodb: database conflict fail

Question

I have a server, and a client.

A client sends a request. The request has a certain key associated with it, e.g. a-1, a-2, b-1, b-4.

If two requests for the same key come in at once, there will be a conflict error, as the same data structures are being modified.

I can adapt the client to simply not send two requests of the same key at once. However, I'd like this system to work with multiple clients, too. It seems silly to have the clients coordinate what they send to the server. Instead, I'd like the server to simply block on a request of a certain key if that key is already being modified, until the other requests with that same key are done.

To this end, I've created a locking system. At the beginning of the function on the server, I do:

key = ...
print "Acquiring %s lock..." % (key,)
KEY_LOCKS[key].acquire()
print "%s lock acquired." % (key,)
def after_commit_hook(success):
    KEY_LOCKS[key].release()
    print "(after %s commit): Released %s lock" % (('failed', 'successful')[success], key)
transaction.get().addAfterCommitHook(after_commit_hook)

where KEY_LOCKS is a dict mapping keys to threading.Locks. Afterwards follows the code that modifies the persistent data structures.

What I assume would happen is that, if a request comes in for a key that's already being processed, it would block when acquiring the lock. Only when the earlier request has already been committed (thus being beyond any conflict errors), would the new request resume. The requests do nothing that would conflict until the lock is acquired.

Most of the requests work fine:

Acquiring a-b lock...
a-b lock acquired.
(after successful commit): Released a-b lock
Acquiring a-c lock...
a-c lock acquired.
(after successful commit): Released a-c lock

However, there is still an issue when the same key is sent, even though the locking seems to work:

Acquiring q-q lock...
q-q lock acquired.
Acquiring q-q lock...
(after successful commit): Released q-q lock
q-q lock acquired.
(after failed commit): Released q-q lock
repoze.retry retrying, count = 1
Traceback (most recent call last):
...
ConflictError: database conflict error (oid 0x13009b, class persistent.list.PersistentList)

And then the request retries. Note that the q-q lock was only acquired after the successful commit.

What gives? Why is this system not preventing conflict errors? Where is my assumption incorrect?

---

EDIT: Well, if, before the transaction.get().addAfterCommitHook(after_commit_hook) line I put transaction.begin(), it works. For the life of me I can't figure out why. Before the transaction.begin() line, the entirety of my code is:

post = request.params
if not post: return Response("No data!")

data = eval(post['data'])
time_parsed = time.time() 
my_app = request.context

This solves my problem but I'm not putting that as an answer 'cause I still want to know: Why does it give conflict errors if I don't start a fresh transaction right before?

Answer 1

The ZODB provides connections with a consistent view from the moment a transaction begins. That means most of all that a thread will not see changes in the database made by other threads until a new transaction begins! This is fundamental feature of databases called Multiversion Concurrency Control.

In other words, if you want your application threads to not conflict by using locking, you also need to start a new transaction when the lock becomes available.

In my opinion, locking is a performance bottleneck that you want to avoid. The longer you hold the lock, the larger the chance you run into inconsistent data (your starting state vs. the database state).

In Zope, an optimistic approach is used instead: when a conflict error is raised, the transaction is aborted, and retried anew, from the start, instead. If your transactions are short, fast affairs, you avoid most locking problems, and instead you just recompute your changes to the database whenever the persistent change has changed to cause a conflict.