Using eventlet with SQLAlchemy
-
20-01-2021 - |
Question
I'm using eventlet to build a simple website crawler starting from this example. I would like to use SQLAlchemy to store web pages' content and metadata for further processing. It's possible to use SQLAlchemy along with eventlet? Could anyone provide a simple example?
Solution
Sorry for late reply.
It would really depend on database connection library you use.
- For C extension library, like MySQLdb, Eventlet has db_pool module as used in OpenStack example. Basically, it creates a pool of OS threads to execute blocking DB operations. So you can't really have a lot of parallel queries, but most probably your database would not survive them anyway, so that's not an issue.
- For pure Python library, like myconnpy, pg8000, then just call
eventlet.monkey_patch()
. - As a special case, Eventlet supports monkey patching for psycopg2. It is a C extension but it is awesome enough to provide IO hooks, so you get best of both worlds. Again, simple call to
monkey_patch()
would do the trick.
OTHER TIPS
The OpenStack Compute (Nova) project uses SQLAlchemy and eventlet, you might be interested in their approach.
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow