Force workers to instantly pick up failed messages when persistent delivery mode is enabled

Question

I have a RabbitMQ Server set up where I fetch messages using Python-Pika. The problem is that if I have persistent delivery mode enabled, and the workers fails to process a message. Instead of releasing the message, it will keep it until the message until the RabbitMQ connection has been reset.

Is there a way to make sure that the message that failed to process automatically gets picked up again within a reasonable time-frame from an available worker, including the same one?

This is my current code

if success:
    ch.basic_ack(delivery_tag=method.delivery_tag)
else:
    syslog.syslog('Error (Callback) -- Failed to process payload: %s' % body)

The idea is that I never want to lose a message, instead I want it to get re-published, or rather picked up again if it failed. This should always be the case until the message has been successfully processed by a worker. This normally happens when one of the worker is unable to open a connection to the HTTP server.

Answer 1

I finally figured out why this was happening. I did not realize that it wasn't enough to simply acknowledge when you were done with a message, but also had to reject any message you could not process using channel.basic_reject. This may seem obvious, but it is not the default behavior for AMQP.

Basically we have to release the message using basic_reject with requeue set to True. The important factor here is the requeue keyword which prevents the message from being discarded, and instead queues it up again, so that one of our available workers can process it.

if success:
    # On Success - Mark message as processed.
    ch.basic_ack(delivery_tag=method.delivery_tag)
else:
    # Else - Mark message as rejected and move it back to the queue.
    ch.basic_reject(delivery_tag=method.delivery_tag, requeue=True)

I found some really useful information in this article, and there are more technical details on the reject keyword in this blog post.