Django, ImportError: cannot import name Celery, possible circular import?

Question

I went through this example here:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

All my tasks are in files called tasks.py.

After updating celery and adding the file from the example django is throwing the following error, no matter what I try:

ImportError: cannot import name Celery

Is the problem possibly caused by the following?

app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

Because it goes through all tasks.py files which all have the following import.

from cloud.celery import app

cloud/celery.py:

from __future__ import absolute_import

import os, sys
from celery import Celery
from celery.schedules import crontab
from django.conf import settings

BROKER_URL = 'redis://:PASSWORD@localhost'

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cloud.settings')
app = Celery('cloud', broker=BROKER_URL)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

if "test" in sys.argv:
    app.conf.update(
        CELERY_ALWAYS_EAGER=True,
    )
    print >> sys.stderr, 'CELERY_ALWAYS_EAGER = True'

CELERYBEAT_SCHEDULE = {
    'test_rabbit_running': {
        "task": "retail.tasks.test_rabbit_running",
        "schedule": 3600, #every hour
    },
    [..]

app.conf.update(
    CELERYBEAT_SCHEDULE=CELERYBEAT_SCHEDULE
)

retail/tasks.py:

from cloud.celery import app

import logging
from celery.utils.log import get_task_logger
logger = get_task_logger('tasks')
logger.setLevel(logging.DEBUG)

@app.task
def test_rabbit_running():
    import datetime
    utcnow = datetime.datetime.now()
    logger.info('CELERY RUNNING')

The error happens, when I try to access a url that is not valid, like /foobar.

Here is the full traceback:

Traceback (most recent call last):
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 126, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 255, in __call__
    response = self.get_response(request)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 178, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 220, in handle_uncaught_exception
    if resolver.urlconf_module is None:
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 342, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/src/slicephone/cloud/cloud/urls.py", line 52, in 
    urlpatterns +=  patterns('', url(r'^search/', include('search.urls')))
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 25, in include
    urlconf_module = import_module(urlconf_module)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/src/slicephone/cloud/search/urls.py", line 5, in 
    from handlers import SearchHandler
  File "/opt/src/slicephone/cloud/search/handlers.py", line 15, in 
    from places import handlers as placeshandler
  File "/opt/src/slicephone/cloud/places/handlers.py", line 23, in 
    import api as placesapi
  File "/opt/src/slicephone/cloud/places/api.py", line 9, in 
    from djapi import *
  File "/opt/src/slicephone/cloud/places/djapi.py", line 26, in 
    from tasks import add_single_place, add_multiple_places
  File "/opt/src/slicephone/cloud/places/tasks.py", line 2, in 
    from cloud.celery import app
  File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in 
    from celery import Celery
  File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in 
    from celery import Celery
ImportError: cannot import name Celery

Answer 1

Adding the following lines to cloud/celery.py:

import celery
print celery.__file__

gave me the file itself and not the celery module from the library. After renaming celery.py to celeryapp.py and adjusting the imports all errors were gone.

Note:

That leads to a change in starting the worker:

celery worker --app=cloud.celeryapp:app

For those running celery==3.1.2 and getting this error:

TypeError: unpack_from() argument 1 must be string or read-only buffer, not memoryview

Apply the patch mentioned here: https://github.com/celery/celery/issues/1637

Answer 2

With Django 1.7.5, Celery 3.1.17, and Python 2.7.6 I found that I was still getting these ImportError: cannot import name Celery. But only when running tests under PyCharm 4.0.4.

I found that a solution was not to rely on from __future__ import absolute_import as described in First Steps with Django. Instead I renamed proj/proj/celery.py to proj/proj/celery_tasks.py and then changed the content of __init__.py to match: from .celery_tasks import app as celery_app. No more multiple instances of files named celery.py to cause import confusion seemed to be a simpler approach.

Answer 3

got the same error

my celery settings filename which was(celery.py) was conflicting with 'celery' package...

so while doing this-> from celery import Celery , it was raising error- cannot import name Celery

solution->just change the 'celery.py' to something else like 'celery-settings.py'

Answer 4

Work for me ( some bug after deploy in server ): Remove all *.pyc files from project and restart him.

Answer 5

Did you add the line:

from __future__ import absolute_import

to the top of your cloud/celery.py module?

Read the breakdown of the example here: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

Answer 6

I got the same error. It turns out there was a problem with my Celery version. I upgraded to 3.1 and celeryd is now deprecated for this version (http://celery.readthedocs.org/en/latest/whatsnew-3.1.html). So I had to downgrade to the version 3.0.19 that was the previous stable version used for the project, and it works good so far.

    pip install celery==3.0.19

Anyway, if you don't want to downgrade, the replacement for celeryd in the version 3.1 is celery worker. Check here for more info: http://celery.readthedocs.org/en/latest/userguide/workers.html.

Hope this helps! :)

Answer 7

For someone who want to know what cause this error:
I have meet this problem just now, then I found the problem --- sys.path.
Maybe you add some path to sys.path like me, I add below code in manage.py,

ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
SRC_PATH = os.path.join(ROOT_PATH, 'src')
CONF_PATH = os.path.join(ROOT_PATH, 'conf')

sys.path.insert(0, SRC_PATH)
sys.path.insert(0, CONF_PATH)

so, from celery import Celery would search celery in SRC_PATH and CONF_PATH first, that's the problem.

change to

sys.path.append(SRC_PATH)
sys.path.append(CONF_PATH)

It would search in python's lib and site-packages first. Solved perfectly.

Answer 8

I have face similar type of issue: from celery import Celery ImportError: cannot import name 'Celery' from 'celery'

Another simple way to solve this: If your package have celery configuration in celery.py this is the reason that it is causing problems. Rename it something like celery_settings.py

Answer 9

Note that older Django projects have the manage.py script in the same directory as the project directory. That is, the structure looks like this:

- proj/
  - proj/__init__.py
  - proj/celery.py
  - proj/urls.py
  - proj/manage.py
  - proj/settings.py

instead of this:

- proj/
  - proj/__init__.py
  - proj/celery.py
  - proj/settings.py
  - proj/urls.py
- manage.py

In this case, you will just have to rename the celery.app file to something different, like celeryapp.py as suggested in the accepted answer above.

Answer 10

I got the same error.

Seems that from __future__ import absolute_import DOES NOT work for Python 2.6.1, still not raising an error.

Upgraded to Python 2.7.5 and it just worked.