Celery worker working from command line but not as daemon, in a virtualenv

Question

System Info

Ubuntu 12.04 LTS

Django 1.5.5

Python 2.7.3

Celery 3.1.9

I am running this on a vagrant virtual machine (with puppet) and attempting to set up celery to run the worker as a daemon as described in the celery docs here as well as the celery setup for django described here. I am using a virtualenv for the project located at

/home/vagrant/virtualenvs/myproj

The actual project files are located at

/srv/myproj

I have been able to start the the worker and the beat scheduler without issue when located in the /srv/myproj directory using the command line statements.

~/virtualenvs/myproj/bin/celery -A app beat
~/virtualenvs/myproj/bin/celery worker -A app

Both beat and the worker start without issue and the scheduled task is passed to the worker and executed. The problem arises when I attempt to attempt to run them as background processes. I am using the scripts found on the celery github repo in /etc/init.d/ and using the following configuration settings in my celeryd and celerybeat files located in /etc/default

CELERY_BIN="/home/vagrant/virtualenvs/myproj/bin/celery"
CELERYD_CHDIR="/srv/myproj"

Attempting to run the services as sudo with

sudo service celeryd start
sudo service celerybeat start

Causes an error message to be thrown, I believe this is because it is using the python located in usr/lib instead of the python in the virtualenv. The error thrown is a cannot import name (the package exists in the virtualenv but not globally hence my assumption).

I also noticed on the Running the worker as a daemon it states that workers should run as unprivileged users, and that you should start workers and beat as using the multi or --detach command. This way I was able to start the worker (not beat) but all the .log and .pid files are being created in my current directory instead of where I've specified in the /etc/default/celeryd config file.

Does anyone have a solution for getting celery to work in a virtualenv? I feel like I'm really close and am overlooking some simple part of the configuration.

Answer 1

I was eventually able to get this working by using supervisor and setting the environment variables in the [program:celery] environment option.