How to Implement Priority Queues in RabbitMQ/pika

Question

I am looking to implement a priority queue with RabbitMQ. The mailing list recommends to use multiple queues, each queue representing a different priority level.

My question is, how do you poll multiple queues in some prioritized order using pika (or possibly some other python library)?

Answer 1

I don't think there is a way to do it naively at the consumer level with pika, as all consumers by default have the same priority.

What I might do to solve the problem would be to have the two queues as suggested on the mailing list, each queue with its own consumer. In the consumer callback for each consumer instead of dealing with the message directly I would put it into a priority queue, then call a function that reads the highest priority message from the queue and handles it.

Another question with a similar response.

Answer 2

The accepted answer is outdated. From rabbitmq 3.5.0 there's native support for priority queues:

RabbitMQ has priority queue implementation in the core as of version 3.5.0. Any queue can be turned into a priority one using client-provided optional arguments

It's also available as of pika 1.1.0

class pika.spec.BasicProperties(content_type=None, content_encoding=None, headers=None, delivery_mode=None, priority=None, correlation_id=None, reply_to=None, expiration=None, message_id=None, timestamp=None, type=None, user_id=None, app_id=None, cluster_id=None)

The code using this feature might look as follows:

channel.basic_publish(properties=pika.BasicProperties(priority=your_priority),
                      exchange=...,
                      routing_key=...,
                      body=...)

Answer 3

In case you stumble into this question after it's been accepted. RabbitMQ has a plugin that allows to set up one queue with priorites: https://github.com/rabbitmq/rabbitmq-priority-queue