Pickling weakref in Python

Question

I am still pretty new to Python and even newer to pickling. I have a class Vertex(ScatterLayout) with a __getnewargs__():

def __getnewargs__(self):
    return (self.pos, self.size, self.idea.text)

My understanding is that this will cause the pickle to pickle the object from __getnewargs__() rather than the object's dictionary.

The pickle is called in the following method (in a different class MindMapApp(App)):

def save(self):
    vertices = self.mindmap.get_vertices()
    edges = self.mindmap.get_edges()

    output = open('mindmap.pkl', 'wb')

    #pickle.dump(edges, output, pickle.HIGHEST_PROTOCOL)
    pickle.dump(vertices, output, pickle.HIGHEST_PROTOCOL)

    output.close()

When I call the save() method I get the following error:

pickle.PicklingError: Can't pickle <type 'weakref'>: it's not found as __builtin__.weakref

What am I missing or not understanding? I have also tried implementing the __getstate__() / __setstate__(state) combination, with the same result.

Answer 1

You definitely can serialize a weakref, and you can serialize a dict and a list. Basically, it matters what the objects contain. If the dict or list contains any unpicklable items, then the pickling will fail. If you want to pickle a weakref, you have to use dill and not pickle. dill extends pickle to include objects that are otherwise unpicklable with pickle. However, note that with dill, an unpickled weakref will deserialize as dead references.

>>> import dill
>>> import weakref
>>> dill.loads(dill.dumps(weakref.WeakKeyDictionary()))
<WeakKeyDictionary at 4528979192>
>>> dill.loads(dill.dumps(weakref.WeakValueDictionary()))
<WeakValueDictionary at 4528976888>
>>> class _class:
...   def _method(self):
...     pass
... 
>>> _instance = _class()
>>> dill.loads(dill.dumps(weakref.ref(_instance)))
<weakref at 0x10d748940; dead>
>>> dill.loads(dill.dumps(weakref.ref(_class())))
<weakref at 0x10e246a48; dead>
>>> dill.loads(dill.dumps(weakref.proxy(_instance)))
<weakproxy at 0x10e246b50 to NoneType at 0x10d481598>
>>> dill.loads(dill.dumps(weakref.proxy(_class())))
<weakproxy at 0x10e246ba8 to NoneType at 0x10d481598>

Answer 2

I worked around this by switching between weak/strong reference in __getstate__/__setstate__:

class DBObject(object):
    def __getstate__(self):
        s = self.__dict__.copy()
        s['_db'] = s['_db']()
        return s

    def __setstate__(self, state):
        self.__dict__ = state.copy()
        self.__dict__['_db'] = weakref.ref(self.__dict__['_db'])