Question

The requirement is as follows:

  • There are N producers, that generate messages or jobs or whatever you want to call it.
  • Messages from each procuder must be processed in order and each message must be processed exactly once.
  • There's one more restriction: at any time for any given producer there must be not more than one message that is being processed.
  • The consuming side consists of a number of threads (they are identical in their functionality) that are spread across a number of processes - it is a WSGI application run via mod_wsgi.

At the moment, the queueing on the consuming side is implemented as a custom queue, that subclasses Queue, but it has its own problems that I won't get into, the main one being that upon process restart its queue is lost.

Is there a product, that will make it possible to fulfill the requirements I've outlined above? Support for persistency would've been great, though that is not so important (since the queue will not reside in the worker process' memory any more).

No correct solution

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top