Celeryd running multiple daemons

Question

I'm attempting to use Celery with the RabbitMQ backend to run asynchronous background tasks on Fedora, and I'm finding it frustratingly unreliable. The biggest problem is that it occasionally runs the identical task multiple times, and I'm having trouble diagnosing why.

I don't think my code's triggering the task multiple times, as I've tested by manually launching the task (which creates a db record) once, and see that two identical records are created. If I run the code from a Django shell, only one record is created, so the problem definitely seems to be Celery.

I have 3 servers each running celeryd, with only the first running rabbitmq, and all 3 servers share the same database. My first through was that there's was a problem in my celery settings causing celeryd on each server to retrieve and execute the same task. However, nothing in the celery docs seems to differentiate between a single host and multi-host setup, so assuming I've specified the correct BROKER_HOST/PORT/USER/PASSWORD/VHOST in my settings.py for each server, it should "just work" with multiple hosts. Is this correct?

The other thought I had was that I might somehow be running multiple instances of celeryd, although I'm not sure how to check for this. Two servers are running Fedora 13, and when I run ps aux | grep .*.py I get:

root     24806  0.1  1.8  51404 31328 ?        Sl   Oct19   9:25 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler
root     24900  0.1  1.6  51404 28592 ?        S    Oct19   6:02 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler
root     24901  0.3  9.4 183232 161948 ?       S    Oct19  22:32 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler

Does this indicate that 3 separate instances of celeryd are running? And if so, is this an error and should I kill 2 of them?

My third server is running Fedora 17, which has a different services framework. When I run systemctl status celeryd.service I get:

celeryd.service - LSB: celery task worker daemon
      Loaded: loaded (/etc/rc.d/init.d/celeryd)
      Active: active (exited) since Fri, 19 Oct 2012 10:59:38 -0400; 4 days ago
     Process: 732 ExecStop=/etc/rc.d/init.d/celeryd stop (code=exited, status=0/SUCCESS)
     Process: 738 ExecStart=/etc/rc.d/init.d/celeryd start (code=exited, status=0/SUCCESS)
      CGroup: name=systemd:/system/celeryd.service

I'm not sure how to interpret this. "Active" usually it's running, but "exited" usually means it's not running. When I run ps aux | grep .*celery.* I only get:

root     25142  0.0  0.0 109400   932 pts/0    S+   11:28   0:00 grep --color=auto .*celery.*

So does this mean celeryd is not running, or should I be looking for something else?

Edit: Based on this answer, I think the 3 processes may be the default.

Answer 1

I posted in the comments, but I'm convinced this is your problem, see here:

root     24806  0.1  1.8  51404 31328 ?        Sl   Oct19   9:25 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler
root     24900  0.1  1.6  51404 28592 ?        S    Oct19   6:02 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler
root     24901  0.3  9.4 183232 161948 ?       S    Oct19  22:32 ../../.env/bin/python manage.py celeryd -f /var/log/myapp/celeryd.log -l WARNING --pidfile /var/run/celeryd.pid -B --scheduler djcelery.schedulers.DatabaseScheduler

Your running celeryd with beat three times. So, your sending beat messages three times.

You should either only one instance with beat on, or (preferably) run celerybeat on its own and take the -B off of the celeryd instances