Minimizing a function with scipy.optimize parallely with ipython

Question

I found a bunch of questions regarding similar issues (this one, for example), and the problem with pickling a closure seems to be all around when dealing with iPython.Parallel but I couldn't get around this particular issue anywhere. So my problem is the following:

I want to solve for the zeros a function f( a, b ) across multiple values of b using an ipcluster. f itself is a complicated function. Let's use a stupid example

import scipy.optimize

def f(a,b):
     return (a+b)**2 + numpy.sin(a*b)

bs = range( 20 )

for b in bs:
     g = lambda a : f(a,b)
     root = scipy.optimize.fsolve( g, 0.0 )
     print root

well, that's a general idea of what it is I'm doing. The thing is, if I try to create a function that returns the root, it will have to create a closure and pass it to scipy.optimize.fsolve (i think). For example, I have tried

 def root( b ):
      g = lambda a : f( a, b )
      return scipy.optimize.fsolve( g, 0.0 )

But if you use iPython.Parallel's map it won't be able to pickle the closures. I couldn't think of any way around that, could you?

---

I guess a minimal example to reproduce the error would be

import numpy
import scipy.optimize

from IPython.parallel import Client

def f( a,b):
    return (a+b)**2+numpy.cos(a*b)

def h( a,b=1.0):
    return (a+b)**2+numpy.cos(a*b)

def root_h( a ):#argument just for mapping, not used
    return scipy.optimize.fsolve( h, 0.0 )

def root(b):
    g = lambda a : f(a,b)
    return scipy.optimize.fsolve( g, 0.0 )

if __name__=='__main__':

    #first normally, this works
    print root( 1.0 )
    print root( 2.0 )

    #now parallely, doesn't work
    c = Client()
    dview = c[:]
    with dview.sync_imports():
        import numpy
        import scipy.optimize

    #this works
    dview.push({'h':h})
    res = dview.map( root_h, [1.0,2.0] )

    for i in res:
        print i

    #this doesn't
    dview.push({'f':f})
    res = dview.map( root, [1.0,2.0] )

    for i in res:
        print i

It throws the error ValueError: Sorry, cannot pickle code objects with closures for me, if anyone can think of a way around it...

Cheers, thanks everyone, SO sure helps a bunch :).

Answer 1

What if you avoid creating a closure in root(b)?

I would try

def root(b):
    g = lambda a, b=b : f(a,b)
    return scipy.optimize.fsolve( g, 0.0 )

which for your example gives the expected result instead of the Pickle Error.

Explanation: I believe the closure occurs because your lambda function was calling f(a,b) where b is a variable from the outer (enclosing) namespace. I you hand in that variable explicitly (lambda;s can have multiple arguments or even keyword arguments) it stops being a closure.