Python 'shelve' that writes itself to disk after each operation

Question

I'm doing a project in Python that involves a lot of interactive work with a persistent dictionary. If I weren't doing interactive development, I could use contextlib.closing and be fairly confident that the shelf would get written to disk eventually. But as it stands there's no chunk of code that I can easily wrap within a 'with' statement. I'd prefer not to have to trust myself to call close() on my shelf at the end of a session.

The amount of data involved is not large, and I would happily sync a shelf to disk after each operation. I've found myself writing a wrapper for shelve that does just that but I'm not a strong enough Python programmer to identify the correct set of dict methods that I'd need to override. And it seems to me that if what I'm doing is a good idea, it's probably been done before. What is the paradigmatically correct way to handle this?

I'll add that I like using shelve because it's a simple module, and it comes with Python. If possible I'd prefer to avoid doing something that requires (for example) pulling in a complicated library for dealing with databases.

I'm using WinXP with SP3, Python 2.7.5 via Anaconda 1.6.2 (32-bit), and running inside Spyder. I can tell by looking at the modified times for the file backing my shelf that the shelf isn't updating until I call sync or close.

Answer 1

What you can do is subclass DbfilenameShelf and override __setitem__ and __delitem__ to automatically sync after each change. Something like this would probably work (untested):

from shelve import DbfilenameShelf

class AutoSyncShelf(DbfilenameShelf):
    # default to newer pickle protocol and writeback=True
    def __init__(self, filename, protocol=2, writeback=True):
        DbfilenameShelf.__init__(self, filename, protocol=protocol, writeback=writeback)
    def __setitem__(self, key, value):
        DbfilenameShelf.__setitem__(self, key, value)
        self.sync()
    def __delitem__(self, key):
        DbfilenameShelf.__delitem__(self, key)
        self.sync()

my_shelf = AutoSyncShelf("myshelf")

I can't vouch for the performance of this, of course.