shelve archives not openable on other computer

Question

I have saved some objects with shelve. In an other file I was able to restore these objects. But when I copy the archive to an other computer shelve gives me a _gdbm.error: File read error. The packages which hold the classes of the stored objects are directly accessible on both computers (but they are stored in different locations and added with PYTHONPATH). Both machines run on Ubuntu 13.10, one is 32bit and the other 64bit.

Shouldn't these archives be machine independent?

On the 64bit machine I get

>>> import shelve
>>> shelve.open('arch.db')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/shelve.py", line 232, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.3/shelve.py", line 216, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.3/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: File read error

and on the 32bit machine it works.

When I create an archive on the 64bit machine, it is openable on the 32bit machine, but the interactive python prompt crashes:

>>> import shelve
>>> s = shelve.open('arch.db')
>>> for i in s.items(): print(i)
...
gdbm fatal: lseek error

I don't even get a traceback.

It is really annoying, I intended to work on both computers, but at the moment I am bound to my slow 32bit eeepc, because I've already saved a lot into the archive.

Answer 1

The problem is that shelve uses gdbm (provided as dbm.gnu as default backend to store the serialized objects. The files created with gdbm depend on the architecture of the system and are therefore only usable on 32bit or 64bit systems.

There are some tools (gdbmexport or gdbm_dump) which allow you to convert the gdbm files, however, the workflow becomes error prone if you want to access the files from both systems on a regular basis.

Fortunately, python provides different backends: dbm.gnu, dbm.ndbm and dbm.dumb. The latter two are platform independent.

import shelve
import dbm
dbm._defaultmod = dbm.ndbm
db = shelve.open('somename')

The database with the above code can be used on 64bit and 32bit systems.

The default backend has to be set only when you create the file. dbm checks the file type of your database before opening and uses the correct backend.

Please note that the above code changes the default dbm for the whole python process. Another component might break if it relies on gdbm being the default.