TypeError for pickle of array of custom objects in Python

Question

I have the following method:

def save(self):
    vertices = self.mindmap.get_vertices()
    edges = self.mindmap.get_edges()

    output = open('mindmap.pkl', 'wb')

    pickle.dump(edges, output)
    pickle.dump(vertices, output)

    output.close()

When I call this method, I get the following error:

Traceback (most recent call last):
  File "main.py", line 72, in <module>
    MindMapApp().run()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/app.py", line 792, in run
    runTouchApp()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/base.py", line 481, in runTouchApp
    EventLoop.window.mainloop()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/core/window/window_pygame.py", line 381, in mainloop
    self._mainloop()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/core/window/window_pygame.py", line 287, in _mainloop
    EventLoop.idle()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/base.py", line 324, in idle
    self.dispatch_input()
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/base.py", line 309, in dispatch_input
    post_dispatch_input(*pop(0))
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/base.py", line 275, in post_dispatch_input
    wid.dispatch('on_touch_up', me)
  File "_event.pyx", line 316, in kivy._event.EventDispatcher.dispatch (kivy/_event.c:4537)
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/uix/behaviors.py", line 110, in on_touch_up
    self.dispatch('on_release')
  File "_event.pyx", line 312, in kivy._event.EventDispatcher.dispatch (kivy/_event.c:4491)
  File "/Applications/Kivy.app/Contents/Resources/kivy/kivy/lang.py", line 1299, in custom_callback
    exec(__kvlang__.co_value, idmap)
  File "./mindmap.kv", line 14, in <module>
    on_release: app.save()
  File "main.py", line 27, in save
    pickle.dump(vertices, output)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1370, in dump
    Pickler(file, protocol).dump(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 600, in save_list
    self._batch_appends(iter(obj))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 615, in _batch_appends
    save(x)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 71, in _reduce_ex
    state = base(self)
TypeError: __init__() takes exactly 0 positional arguments (1 given)

I do not get this error if both arrays are empty. When the above was generated edges was empty and vertices had a size of one. I really don't know how to interpret this error, and as such have no idea how to resolve it.

This is my Vertex class __init__:

def __init__(self, pos=None, size=None):
    super(Vertex, self).__init__()

    self.pos = pos
    self.size = size

    if self.height < self.MINIMUM_HEIGHT + self.TEXT_OFFSET:
        self.height = self.MINIMUM_HEIGHT + self.TEXT_OFFSET

    if self.width < self.MINIMUM_WIDTH + self.TEXT_OFFSET:
        self.width = self.MINIMUM_WIDTH + self.TEXT_OFFSET

    self.originalSize = self.size
    self.originalWidth = self.size[0]

    self.newIdea = TextInput(
        height = self.MINIMUM_HEIGHT,
        size_hint = (None, None),
        pos_hint = {'center_x':.5, 'center_y':.5},

        background_active = '',
        background_normal = '',

        use_bubble = True,

        foreground_color = (.4, .4, .4, 1),
        cursor_color = (.4, .4, .4, 1))
    self.newIdea.bind(text=self.on_text)

    self.idea = Label(
        height = self.MINIMUM_HEIGHT,
        size_hint = (None, None),
        pos_hint = {'center_x':.5, 'center_y':.5},
        color = (.4, .4, .4,1))

    Logger.info('Vertex: created')

Answer 1

Indeed, the Kivy EventDispatcher object is at fault here; the object.__reduce_ex__ method searches for the first baseclass that is not a custom class (it does not have the Py_TPFLAGS_HEAPTYPE flag set), to call __init__ on that class. This fails for that base class, as it's __init__ method doesn't support passing in a single argument.

The work-around is to use the latest pickle protocol instead; it uses a different codepath:

pickle.dump(edges, output, pickle.HIGHEST_PROTOCOL)
pickle.dump(vertices, output, pickle.HIGHEST_PROTOCOL)

because the latest pickle protocol (2) supports new-style classes much more efficiently.

Seeing the complexity of the Kivy object hierarchy, you may need to customise the pickling process.

I'd look into implementing the __getnewargs__ method for example:

def __getnewargs__(self):
    return (self.pos, self.size)

would ensure that just that data is pickled, nothing more. On unpickling, these two values are then used to create new Vertex objects. Note that for this method to be used you have to use the latest protocol!

You could also implement a __getstate__ method; if you return a dictionary, no mirror __setstate__ method is required, but would allow you to initialise some more attributes.

If neither is implemented, the contents of the instance __dict__ is pickled instead. Make sure that everything contained can be pickled.