Python: pickle derived str object

Question

Running:

import pickle

class Foo(str):
    def __new__(cls, s, bar):
        self = super(Foo, cls).__new__(cls, s)
        self.bar = bar
        return self

with open('foo.pkl', 'wb') as w:
    x = pickle.dump(Foo('', 1), w, pickle.HIGHEST_PROTOCOL)

with open('foo.pkl', 'rb') as w:
    x = pickle.load(w)

Results in this exception:

Traceback (most recent call last):
  File "pkl.py", line 13, in <module>
    x = pickle.load(w)
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1083, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: __new__() takes exactly 3 arguments (2 given)

It works, if I remove pickle.HIGHEST_PROTOCOL, but I'd rather use the new protocol, if possible.

I'm just not getting the pickle protocol description. Can anyone help me out? There must be some way with magic methods...

Answer 1

This works without the HIGHEST_PROTOCOL because the older protocols don't use the __new__ constructor for unpickled objects.

But the basic is that you are using the serializing methods of str while you are adding a new argument to __new__. The pickle module nor the builtin str type has any knowledge of this and it will result in the error shown.

To fix it you should define the __getnewargs__ method and return (self.__class__, self.bar) which should be sufficient for it to work again.

Documentation on the special methods used by the pickle module can be found here