multiple workers and multiple queues on celery django daemon

Question

I am using celery-django to queue tasks on my site backend. I am trying to create a setup where I have two queues named "low" and "high" and two workers W1 and W2. I want them to consume the tasks from the queue in the following way:

W1 <-- low, high

W2 <-- high

Normally it can be done like this.

Open terminal 1 and enter $celery worker -n W1 -Q low,high

Open terminal 2 and enter $celery worker -n W2 -Q high

However I am trying to do the same via celeryd daemon.

I am following the steps given in the link: http://celery.readthedocs.org/en/latest/tutorials/daemonizing.html#example-configuration But the available options don't seem enough to fit the requirement.

Please help me with some configs that I am unaware of which could make it possible. I would prefer not to run multiple daemons or use additional tools like supervisord unless really necessary (Maybe you could advice me on this as well).

Answer 1

You can use CELERYD_OPTS option passing -Q parameter similar to those examples from Celery reference:

# Advanced example starting 10 workers in the background:
#   * Three of the workers processes the images and video queue
#   * Two of the workers processes the data queue with loglevel DEBUG
#   * the rest processes the default' queue.
$ celery multi start 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

# You can show the commands necessary to start the workers with
# the 'show' command:
$ celery multi show 10 -l INFO -Q:1-3 images,video -Q:4,5 data
    -Q default -L:4,5 DEBUG

Answer 2

You can specify multiple nodes in CELERY_NODES and pass node names to CELERYD_OPTS arguments, for example:

CELERY_NODES="W1 W2"
CELERYD_OPTS="-Q:W1 low,high -Q:W2 high"