Problematic function pickling

Question

In one module, I have a class which requires a function as one of its parameters:

class Foo():
    def __init__(self, fun):
        self.fun = fun

    def call_fun(self, arg):
        self.fun(arg)

In another module, I have a couple of functions that I eventually pass as parameters when creating this class' objects. I'm binding values in the following way to easily customize the functions:

def sample_function(value):
    return (lambda arg: arg.effect(value))

When initialising Foo objects, I pass the values like this:

foo = Foo(sample_function(5))

Later, I want to shelve some objects, including Foo objects, which quickly turned out impossible (as pickle module returns 'TypeError: can't pickle function objects'). I'm fully aware of the limitations of pickle's serialising mechanism, and I realise why pickling these objects is not possible. Instead I want to redesign my class, or use other means of serialising. How do I redesign this so that I can shelve Foo objects, while retaining their properties?

Answer 1

The main problem with your code is the closuse in "sample_function". Without the closure, it can be done with marshal and types modules, here's some example:

import pickle
import marshal
import types

class Foo():
    def __init__(self, fun):
        self.fun = fun

    def call_fun(self, arg):
        self.fun(arg)

    def save(self, f):
        saved = self.fun
        self.fun = marshal.dumps(self.fun.func_code)
        pickle.dump(self, f)
        self.fun = saved

    @staticmethod
    def load(f):
        foo = pickle.load(f)
        foo.fun = types.FunctionType(marshal.loads(foo.fun), globals())
        return foo

def sample_function(arg):
    arg.effect(4)

class Arg():
    def effect(self, value):
        print "ok: " + str(value)

With this classes and function, you can save:

foo = Foo(sample_function)

f = open('foo', 'w')
foo.save(f)
f.close()

and load:

f = open('foo', 'r')
foo = Foo.load(f)
f.close()

The idea is to marshal the "fun" before pickling (and restore it, so the class is not broken) and to load the marshaled code as a function when loading.

But that won't works with closures (because the closure needs variables from the calling function).