RabbitMQ / Celery with Django hangs on delay/ready/etc - No useful log info

Question

So I just setup celery and rabbitmq, created my user, setup the vhost, mapped the user to the vhost, and ran the celery daemon succesfully (or so I assume)

(queuetest)corky@corky-server:~/projects/queuetest$ ./manage.py celeryd
 celery@corky-server v0.9.5 is starting.
 Configuration ->
. broker -> amqp://celery@localhost:5672/
. queues ->
. celery -> exchange:celery (direct) binding:celery
. concurrency -> 2
. loader -> celery.loaders.djangoapp
. logfile -> [stderr]@WARNING
. events -> OFF
. beat -> OFF

Celery has started.

I created a user of "celery" because I wasn't feeling very inventive in this case.

When I try to do one of the simple examples within the celery docs:

>>> from tasks import add
>>> r = add.delay(2, 2)
>>> r
<AsyncResult: 16235ea3-c7d6-4cce-9387-5c6285312c7c>
>>> r.ready()
(hangs for eternity.)

So I checked the FAQ wondering what else could be up and it told me this is a common bug due to user permissions, so I triple checked those, nothing, made another new user, still nothing. If I import DjangoBrokerConnection from carrot.connection and get the information, it matches up with what's in my celery settings. The FAQ stated to check your log file.

My rabbit.log file isn't very helpful in this situation, simply showing:

=INFO REPORT==== 26-Jan-2010::11:58:22 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:60572

=INFO REPORT==== 26-Jan-2010::11:58:22 ===
starting TCP connection <0.1120.0> from 127.0.0.1:60572

And so forth. At this point, I'm at a loss as to what else my problem could be. I'm running Ubuntu Jaunty and installed RabbitMQ from apt-get.

Thanks in advance for any help.

Answer 1

I just fixed a really ugly bug that would only show up for new users that would have caused this. (http://github.com/ask/celery/commit/a9c1316b15055b67ee3c38d294461fa82ed6d2b5)

Please pull from the master branch at github. If it still doesn't work you probably have to stop rabbitmq, remove the rabbitmq database directory (usually /var/lib/rabbitmq) and start rabbitmq again)

Really sorry for the inconvenience. The bug happened because we recently changed the name of the consumers routing key option to "binding key", but the amqp libraries still use routing_key and we forgot to rewrite the option.

Answer 2

For anyone stumbling upon this: it really does seem to help to remove your /var/lib/rabbitmq, even if the problem seems to go away with updating celery. I was seeing lots of unreliability and unpredictability until I did so.