Bizarre behavior trying to store a deque in a Shelve

Question

I will let the following terminal session speak for itself:

>>> import shelve
>>> s = shelve.open('TestShelve')
>>> from collections import deque
>>> s['store'] = deque()
>>> d = s['store']
>>> print s['store']
deque([])
>>> print d
deque([])
>>> s['store'].appendleft('Teststr')
>>> d.appendleft('Teststr')
>>> print s['store']
deque([])
>>> print d
deque(['Teststr'])

Shouldn't d and s['store'] point to the same object? Why does appendleft work on d but not on s['store']?

Answer 1

It turns out they're not the same so any operations you perform on them won't match:

>>> import shelve
>>> s = shelve.open('TestShelve')
>>> from collections import deque
>>> s['store'] = deque()
>>> d = s['store']
>>> id(s['store'])
27439296
>>> id(d)
27439184

To modify items as you coded, you need to pass the parameter writeback=True:

s = shelve.open('TestShelve', writeback=True)

See the documentation:

If the writeback parameter is True, the object will hold a cache of all entries accessed and write them back to the dict at sync and close times. This allows natural operations on mutable entries, but can consume much more memory and make sync and close take a long time.

You can also do it with writeback=False but then you need to write the code exactly as in the provided example:

# having opened d without writeback=True, you need to code carefully:
temp = d['xx']      # extracts the copy
temp.append(5)      # mutates the copy
d['xx'] = temp      # stores the copy right back, to persist it

Answer 2

shelve is pickleing (serializing) the object. Of necessity, this makes a copy. So, the objects you get back from shelve won't have the same identity as the ones you put in, though they will be equivalent.

If it's important, you could write a deque subclass that automatically re-shelves itself whenever it's modified, although this would probably have poor performance in many use cases.