Celery Tutorial Works Correctly in standard iPython shell but not when using Django's embedded iPython shell

Question

I've been setting up a test instance of django-celery and going through some of the basic celery examples and came across something that seems odd.

I first went through the "First steps with Django" celery page here: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

Everything worked correctly, as did the basic examples in the first few sections of the standard celery tutorial.

python manage.py shell

Once I started trying out some of the canvas primitives in the Django shell I get a NameError, as follows:

$ python manage.py shell
In [1]: from celerytest.tasks import add
In [2]: from celery import group
In [3]: group(add.s(i, i) for i in xrange(10))().get()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <module>()
----> 1 group(add.s(i, i) for i in xrange(10))().get()
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/canvas.pyc in __call__(self, *partial_args, **options)
    397
    398     def __call__(self, *partial_args, **options):
--> 399         tasks = [task.clone() for task in self.tasks]
    400         if not tasks:
    401             return
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/utils/functional.pyc in __iter__(self)
    286
    287     def __iter__(self):  # needed for Python 2.5
--> 288         return iter(self.data)
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/kombu/utils/__init__.pyc in __get__(self, obj, type)
    292             return obj.__dict__[self.__name__]
    293         except KeyError:
--> 294             value = obj.__dict__[self.__name__] = self.__get(obj)
    295             return value
    296
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/celery/utils/functional.pyc in data(self)
    283     @cached_property
    284     def data(self):
--> 285         return list(self.__it)
    286
    287     def __iter__(self):  # needed for Python 2.5
/Users/jacinda/envs/testproj/lib/python2.7/site-packages/django/core/management/commands/shell.pyc in <genexpr>((i,))
----> 1 group(add.s(i, i) for i in xrange(10))().get()
NameError: global name 'add' is not defined

But this next line works:

In [4]: add.delay(2,2).get()
Out[4]: 4

As does explicitly listing subtasks:

In [5]: group([add.s(1,2), add.s(3,4)])().get()
Out[5]: [3, 7]

---

ipython

If I use a regular Python shell and manually import settings instead of using manage.py shell, everything works. I.e.

$ ipython
In [1]: from testproj import settings
In [2]: from celerytest.tasks import add
In [3]: from celery import group
In [4]: group(add.s(i, i) for i in xrange(10))().get()
Out[4]: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

I've looked at the failing case with epdb and can't figure out exactly what's going on.

I've Googled around for this, and the only 100% similar reference I found was this issue on celery's github page: https://github.com/celery/celery/issues/1222. The poster mentions the same issues I'm having, but only says his "Python/Django shell is doing something strange" and doesn't elaborate further. Any ideas on what causes this or if there's somewhere else I should be looking?

Answer 1

After a significant amount of digging, I tracked down the source of the problem and it's actually already fixed on the Django trunk (it's not in the current 1.5 release, though, which is what I'm using).

It turns out that Django's shell command starts up iPython like this (in django/core/management/commands/shell.py)

from IPython import embed
embed()

However, calling embed() inside of a function causes iPython to start with separate local and global namespaces (iPython Github Issue 62). So lambda functions and generator expressions (like group) fail.

To fix this, iPython provides a different way of starting the shell that doesn't have this side effect.

from IPython.frontend.terminal.ipapp import TerminalIPythonApp
app = TerminalIPythonApp.instance()
app.initialize(argv=[])
app.start()

Diff on Django's github site

This should show up in an upcoming version (Django Issue). I tested with this patch applied to 1.5.1 and my code from the original works correctly when it's applied.

Answer 2

This also happens to me but only when using ipython/django shell. If Django uses the default shell or bpython everything works well.

So if you want to bypass the problem for now I advise you to use an alternative shell (I use bpython). If you are using a virtualenv:

$ pip install bpython