Defer processing in Python

Question

I'm trying to write events in a non-blocking fashion so as not to slow down any of our existing processes. It seems like the two options available are:

• use Twisted's defer object
• create a python logging handler

Are there other options. Anybody have experience with this?

Background: We're planning to write events to Amazon's CloudWatch service and I'm concerned about latency from doing the PUT request. I'm not actually so concerned about losing a few PUTs if that changes the answer (we're writing the events for alerting purposes and they'll all be purged after a week anyway).

Answer 1

If logging fits your application, and it sounds so, it should be simpler than using twisted. You could log to a thread safe queue in memory and have a separate thread (or several) pull from it and push to the cloud.

That said, twisted may be faster or at least more scalable, but, especially if you don't know it, with a steeper learning curve.

If the threads approach hits limits you can always switch your logging handler to use twisted, without changing your interface to the app, so that's how I would start, with a simple approach and bases covered.

Answer 2

This would be trivial with gevent (just do the PUT in a separate greenlet and have socket patched).