Tornado + Momoko. Restart corrupted connection

Question

We're using tornado 3.2 + momoko 1.1 + pgpool.

After restarting pgpool - we have to restart tornado.

If we don't restart tornado - we have errors like these:

[E 140322 19:23:32 web:1305] Uncaught exception POST /api/user/balance/ (127.0.0.1)
    HTTPRequest(protocol='http', host='127.0.0.1:3001', method='POST', uri='/api/user/balance/', version='HTTP/1.0', remote_ip='127.0.0.1', headers={'Host': '127.0.0.1:3001', 'Connection': 'close', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': '55', 'Accept-Encoding': 'gzip'})
    Traceback (most recent call last):
      File "/projects/application_new/env/lib/python2.7/site-packages/tornado/web.py", line 1192, in _stack_context_handle_exception
        raise_exc_info((type, value, traceback))
      File "/projects/application_new/env/lib/python2.7/site-packages/tornado/web.py", line 1375, in wrapper
        result = method(self, *args, **kwargs)
      File "api_server.py", line 435, in post
        UserBalanceForJSON(self.db, user_id=user_id).results(self.json_callback)
      File "/projects/application_new/env/lib/python2.7/site-packages/tornado/gen.py", line 159, in wrapper
        runner.run()
      File "/projects/application_new/env/lib/python2.7/site-packages/tornado/gen.py", line 529, in run
        yielded = self.gen.throw(*exc_info)
      File "/projects/application_new/api/models/orm.py", line 16, in results
        result = yield momoko.Op(self.db.execute, sql, [])
      File "/projects/application_new/env/lib/python2.7/site-packages/tornado/gen.py", line 520, in run
        next = self.yield_point.get_result()
      File "/projects/application_new/env/lib/python2.7/site-packages/momoko/utils.py", line 41, in get_result
        raise error
    ProgrammingError: execute cannot be used while an asynchronous query is underway

I've found that django users have same troubles, when using greenlets.

How to avoid such situation without tornado restart?

Answer 1

Temporary solution - close connection, if error happened in it. For it, let's create class inspired by momoko.Op:

class OpWithFallback(gen.Task):
    """
    Run a single asynchronous operation.

    Behaves like `tornado.gen.Task`_, but raises an exception (one of Psycop2's
    exceptions_) when an error occurs related to Psycopg2 or PostgreSQL.

    Closes connection with error.

    .. _exceptions: http://initd.org/psycopg/docs/module.html#exceptions
    .. _tornado.gen.Task: http://www.tornadoweb.org/documentation/gen.html#tornado.gen.Task
    """
    def get_result(self):
        (result, error), _ = super(OpWithFallback, self).get_result()
        if error:
            self.fallback(error, result)
        return result

    def fallback(self, error, result):
        log.warning('Closing dead connection or connection with idle transaction: %s' % result.connection)
        result.connection.close()
        raise error

And let's use it like:

result = yield OpWithFallback(self.db.execute, sql, [])