The arguments to Pool.map
must be picklable. Module-level functions are picklable, but partial(f, y=y)
is not defined at the module-level and so is not pickable.
There is a simple workaround:
def g(x, y=y):
return f(x, y)
p.map(g, x)
Functions made with functools.partial
used to be unpickable.
However, with Python2.7 or better, you can also define g
(at the module level) using functools.partial:
import multiprocessing as mp
import functools
def f(x, y):
return x**2 + y
x = [1,2,3]
y = 10
g = functools.partial(f, y=y)
if __name__ == '__main__':
p = mp.Pool()
print(p.map(g, x))
yields [11, 14, 19]
. But note to get this result f
had to be defined with def
rather than lambda
. I think this is because pickle
relies on "fully qualified" name references to look up function object values.