Cannot get django celeryd to work on Windows

Question

I have a django project which runs just fine with runserver located in:

D:\EverTabs\work\evertabs-website\src\evertabs\

My settings.py and manage.py reside in the above path.

I open the trusty cmd shell, cd to that directory and run manage.py celeryd, I get an exception with the following error:

ImportError: Could not import settings 'evertabs.settings' (Is it on sys.path?): No module named evertabs.settings

fair, enough. Google and StackOverflow to the rescue and I find an answer. I need to provide --settings=settings to the command. Strange, as runserver does not need --settings=settings, but such is life. I'm willing to compromise. I continue to get an error:

Traceback (most recent call last):
  File "C:\Progra~1\Python26\lib\multiprocessing\process.py", line 231, in _bootstrap
    self.run()
  File "C:\Progra~1\Python26\lib\multiprocessing\process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "c:\program files\python26\lib\site-packages\celery-2.4.6-py2.6.egg\celery\concurrency\processes\pool.py", line 173, in worker
    initializer(*initargs)
  File "c:\program files\python26\lib\site-packages\celery-2.4.6-py2.6.egg\celery\worker\__init__.py", line 64, in process_initializer
    app.loader.init_worker()
  File "c:\program files\python26\lib\site-packages\celery-2.4.6-py2.6.egg\celery\loaders\base.py", line 100, in init_worker
    self.on_worker_init()
  File "c:\program files\python26\lib\site-packages\django_celery-2.4.2-py2.6.egg\djcelery\loaders.py", line 92, in on_worker_init
    autodiscover()
  File "c:\program files\python26\lib\site-packages\django_celery-2.4.2-py2.6.egg\djcelery\loaders.py", line 114, in autodiscover
    for app in settings.INSTALLED_APPS])
  File "c:\program files\python26\lib\site-packages\django_celery-2.4.2-py2.6.egg\djcelery\loaders.py", line 124, in find_related_module
    app_path = importlib.import_module(app).__path__
  File "build\bdist.win32\egg\importlib\__init__.py", line 37, in import_module
ImportError: No module named evertabs

So I scratch my head vigorously and decide that maybe if I feed this guy a different pythonpath, he will place nicely. My next try is:

manage.py celeryd --pythonpath=D:\EverTabs\work\evertabs-website\src\

Now we're getting somewhere! No errors. However, my celeryconfig.py, which is in the same directory as settings.py, is not getting processed. None of my tasks are loaded. I went as far as putting a syntax error into celeryconfg.py to prove to myself it is not being loaded.

The next step I did was to use a manage.py from django 1.4. This manage.py is placed in D:\EverTabs\work\evertabs-website\src\ as opposed to D:\EverTabs\work\evertabs-website\src\evertabs\. At this point, I can execute celeryd with the simple syntax and not have to add to the pythonpath. Still, my celeryconfig.py is left untouched:

D:\EverTabs\work\evertabs-website\src>manage.py celeryd

Further research shows that if I add a tasks.py to my top level dir 9where setting.py is located). Celery will auto-discover them. This has worked. however, celeryconfig.py is still not being processed so I cannot do more intricate config in there. For now, this is not a concern, as my tasks are finally found. However, it bothers me greatly that celeryconfig.py is not being processed.

It is now dawning on me that maybe modern celery is not looking for celeryconfig.py after all. I can simply put these settings in my settings.py and celery will pick them up. Things are starting to look good.

Answer 1

django-celery uses settings.py as a celery configuration file.

The other main difference is that configuration values are stored in your Django projects’ settings.py module rather than in celeryconfig.py.

http://ask.github.com/django-celery/introduction.html

Answer 2

Short summary of things I did to get celery to work:

1. Use either --pythonpath to point to the directory above your project directory so that celery can import project.settings or move your manage.py up one level and use the manage.py which django 1.4 generates.

2. Create a tasks.py in your top level of your project (where settings.py is located) and celery will auto-discover them. If you want to use another module, add CELERY_IMPORTS = ("yourproject.anothermodule", ) to your settings.py so that celery will look inside of anothermodule for tasks.

3. Forget about celeryconfig.py which is mentioned in some documentation. Just put all the settings right in your settings.py