IPython Parallel LoadBalancedView GIL

Question

I use a loadbalancedview from Ipython.parallel to call a function on an iterable, ie

from IPython.parallel import Client
from functools import partial

rc = Client()
lview = rc.load_balanced_view()
lview.block = True

def func(arg0, arg1, arg2):
    return func2(arg0) + arg1 + arg2

def func2(arg0):
    return 2*arg0

answer = lview.map(partial(func, arg1=A, arg2=B), iterable_data)

Does the fact that func calls func2 make func not be executed in parallel (ie. does the GIL come into play?) I assume that when you call map, each cluster node gets a copy of func, but do they also get copies of func2. Further, does the fact that I use functools.partial cause any problems?

Answer 1

Does the fact that func calls func2 make func not be executed in parallel (ie. does the GIL come into play?)

Not at all. The GIL is not at all relevant here, nor is it ever relevant in the parallelism in IPython.parallel. The GIL only comes up when coordinating threads within each engine, or within the Client process itself.

I assume that when you call map, each cluster node gets a copy of func, but do they also get copies of func2.

It should, but this is actually where your code will have a problem. IPython does not automatically track closures, and code dependencies in the interactive namespace, so you will see:

AttributeError: 'DummyMod' object has no attribute 'func'

This is because partial(func, arg1=A, arg2=B) contains a reference to __main__.func, not the code of the local func itself. When the partial arrives on the engine, it is deserialized, and __main__.func is requested, but undefined (__main__ is the interactive namespace on the engine). You can address this simply by ensuring that func and func2 are defined on the engines:

rc[:].push(dict(func=func, func2=func2))

At which point, your map should behave as expected.

If you instruct IPython to use the enhanced pickling library dill it will get closer to not having to manually send references, but it doesn't cover every case.