Simultaneously modify different keys in ZODB

Question

I'm using ZODB as a persistent storage for objects that are going to be modified through a webservice. Below is an example to which I reduced the issue. The increment-function is what is called from multiple threads. My problem is, that when increment is called simultaneously from two threads, for different keys, I'm getting the conflict-error.

I imagine it should be possible to resolve this, at least as long different keys are modified, in a proper way? If so, I didn't manage to find an example on how to... (the zodb-documentation seems to be somewhat spread across different sites :/ )

Glad about any ideas...

import time
import transaction
from ZODB.FileStorage import FileStorage
from ZODB.DB import DB
from ZODB.POSException import ConflictError

def test_db():
    store = FileStorage('zodb_storage.fs')
    return DB(store)

db_test = test_db()     

# app here is a flask-app
@app.route('/increment/<string:key>')
def increment(key):
    '''increment the value of a certain key'''

    # open connection
    conn = db_test.open()
    # get the current value:
    root = conn.root()
    val = root.get(key,0)    

    # calculate new value 
    # in the real application this might take some seconds
    time.sleep(0.1)
    root[key] = val + 1     

    try:
        transaction.commit()
        return '%s = %g' % (key, val)
    except ConflictError:
        transaction.abort()
        return 'ConflictError :-('

Answer 1

You have two options here: implement conflict resolution, or retry the commit with fresh data.

Conflict resolution only applies to custom types you store in the ZODB, and can only be applied if you know how to merge your change into the newly-changed state.

The ZODB looks for a _p_resolveConflict() method on custom types and calls that method with the old state, the saved state you are in conflict with, and the new state you tried to commit; you are supposed to return the merged state. For a simple counter, like in your example, that'd be a as simple as updating the saved state with the change between the old and new states:

class Counter(Persistent):
    def __init__(self, start=0):
        self._count = start

    def increment(self):
        self._count += 1
        return self._count

    def _p_resolveConflict(self, old, saved, new):
        # default __getstate__ returns a dictionary of instance attributes
        saved['_count'] += new['_count'] - old['_count']
        return saved

The other option is to retry the commit; you want to limit the number of retries, and you probably want to encapsulate this in a decorator on your method, but the basic principle is that you loop up to a limit, make your calculations based on ZODB data (which, after a conflict error, will auto-read fresh data where needed), then attempt to commit. If the commit is successful you are done:

max_retries = 10
retry = 0

conn = db_test.open()
root = conn.root()

while retry < max_retries:
    val = root.get(key,0)    
    time.sleep(0.1)
    root[key] = val + 1

    try:
        transaction.commit()
        return '%s = %g' % (key, val)
    except ConflictError:
        retry += 1

raise CustomExceptionIndicatingTooManyRetries