How to make a decorator that handles failover with motor and tornado?

Question

I am trying to write a decorator, that takes a function which interacts with mongodb and if an exception occurs it retries the interaction. I have the following code:

def handle_failover(f):
    def wrapper(*args):
        for i in range(40):
            try:
                yield f(*args)
                break
            except pymongo.errors.AutoReconnect:
                loop = IOLoop.instance()
                yield gen.Task(loop.add_timeout, time.time() + 0.25)
    return wrapper


class CreateHandler(DatabaseHandler):
    @handle_failover
    def create_counter(self, collection):
        object_id = yield self.db[collection].insert({'n': 0})
        return object_id

    @gen.coroutine
    def post(self, collection):
        object_id = yield self.create_counter(collection)
        self.finish({'id': object_id})

But this doesn't work. It gives an error that create_counter yields a generator. I've tried making all the functions @gen.coroutines and it didn't help.

How can I make handle_failover decorator work?

edit: No decorators for now. This should create a counter reliably and return object_id to the user. If exception is raised 500 page gets displayed.

class CreateHandler(DatabaseHandler):
    @gen.coroutine
    def create_counter(self, collection, data):
        for i in range(FAILOVER_TRIES):
            try:
                yield self.db[collection].insert(data)
                break
            except pymongo.errors.AutoReconnect:
                loop = IOLoop.instance()
                yield gen.Task(loop.add_timeout, time.time() + FAILOVER_SLEEP)
            except pymongo.errors.DuplicateKeyError:
                break
        else:
            raise Exception("Can't create new counter.")

    @gen.coroutine
    def post(self, collection):
        object_id = bson.objectid.ObjectId()
        data = {
            '_id': object_id,
            'n': 0
        }
        yield self.create_counter(collection, data)
        self.set_status(201)
        self.set_header('Location', '/%s/%s' % (collection, str(object_id)))
        self.finish({})

Although I still don't know how to make increment of the counter idempotent because the trick with DuplicateKeyError is not applicable here:

class CounterHandler(CounterIDHandler):
    def increment(self, collection, object_id, n):
        result = yield self.db[collection].update({'_id': object_id}, {'$inc': {'n': int(n)}})
        return result

    @gen.coroutine
    def post(self, collection, counter_id, n):
        object_id = self.get_object_id(counter_id)
        if not n or not int(n):
            n = 1
        result = yield self.increment(collection, object_id, n)
        self.finish({'resp': result['updatedExisting']})

Answer 1

You most likely don't want to do this. Better to show an error to your user than to retry an operation.

Blindly retrying any insert that raises AutoReconnect is a bad idea, because you don't know if MongoDB executed the insert before you lost connectivity or not. In this case you don't know whether you'll end up with one or two records with {'n': 0}. Thus you should ensure that any operation you retry this way is idempotent. See my "save the monkey" article for detailed information.

If you definitely want to make a wrapper like this, you need to make sure that f and wrapper are both coroutines. Additionally, if f throws an error 40 times you must re-raise the final error. If f succeeds you must return its return value:

def handle_failover(f):
    @gen.coroutine
    def wrapper(*args):
        retries = 40
        i = 0
        while True:
            try:
                ret = yield gen.coroutine(f)(*args)
                raise gen.Return(ret)
            except pymongo.errors.AutoReconnect:
                if i < retries:
                    i += 1
                    loop = IOLoop.instance()
                    yield gen.Task(loop.add_timeout, time.time() + 0.25)
                else:
                    raise
    return wrapper

But only do this for idempotent operations!