How to pickle a callback chain

Question

I have a user-defined state machine in Twisted. The user can define handlers for different state changes, which I implement by using a Twisted deferred that I let them add callbacks to. Whenever I move from one state to another, I simply fire the appropriate deferred.

One of the project requirements is the ability to save this state machine to disk, along with all its callbacks. I thought I could simply pickle the state machine and I'd be done, but I get a PickleError when I try to serialize user-defined functions.

Does anybody know of a way to serialize functions? The error is reproduced in the below code sample:

import pickle
from twisted.internet.utils import defer

def foo(*args):
  def bar():
    print args
  return bar

d = defer.Deferred()
d.addCallback(foo("Hello", "world"))
pickle.dumps(d)

This last line gives the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/pickle.py", line 1366, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.5/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 725, in save_inst
    save(stuff)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "/usr/lib/python2.5/pickle.py", line 663, in _batch_setitems
    save(v)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/usr/lib/python2.5/pickle.py", line 615, in _batch_appends
    save(x)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 562, in save_tuple
    save(element)
  File "/usr/lib/python2.5/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.5/pickle.py", line 748, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <function bar at 0xb753fe2c>: it's not found as __main__.bar

Are there any solutions to this? Maybe I need to restrict the types of functions that users can add as callbacks?

Thanks,
Jonathan

Answer 1

Don't try to pickle Deferreds. It's not supported by Twisted. Even if you manage to construct something that seems to work (and it's not entirely impossible), a later release of Twisted might break all of your saved state.

Deferreds are for controlling the flow of events through your code. They're not for storing application state. If you want to persist your application state, separate it from any Deferreds and serialize just it.

When you do this, you probably also want to avoid using pickle for the serialization format. Pickle is not a good way to store data. It is a high complex format which is very sensitive to changes in Python versions and library versions. It has no means to define a schema, so you can never really be sure what you're serializing or what you have serialized. It is very difficult to inspect a pickle separately from loading it, so if it ever breaks (as it will if you decide to rename a class which you have pickled instances of), recovering the data is a major hassle.

Answer 2

Replace the foo/bar functions with a callable class instance:

class foo(object):
    def __init__(self, *args):
        self.args = args
    def __call__(self):
        print self.args

d = defer.Deferred()
d.addCallback(foo("Hello", "world"))
pickle.dumps(d)