Python: concurrent access (read/write) of network file

Question

With Python (I am using v3.4), I am creating a program that will run on up to 20 machines concurrently connected to a same network. Those machines will need from time to time to access the network and make some operations (create/delete/modify files)…

What I had currently planned was:

• network will have a folder "db"
• I will use the shelve module to have my shared data (a simple dictionary of less than 3,000 entries containing some path names and some logging information) in this folder. Since there is not so much data it is a convenient module that will be fast enough
• to avoid operations by multiple machines on the network at the same time (and avoid conflicts), machines will have to lock the shelve file on the network before doing anything (anyway they will also need to update this file) and will unlock only at the end of their operation (updating its contents if necessary)

The problem I have is that shelve does not have a convenient way for concurrent access or locking mechanism. I understand that my 2 possibilities are:

• either I don't use shelve and use another module to manage my simple database (ex: sqlite3 except it is a bit more complex than the simple shelve module)
• either I create a locking mechanism that will work on network by multiple machines (so far I didn't find a module that seemed totally reliable)

Additional requirements (if possible) are:

• It will mainly be used with Windows but I would like the solution to be cross-platform so that I can re-use it with Linux
• network file system will be anything accessible through a standard explorer (in linux/windows) through an address like "\Machine\Folder". It could be either a file server or just a shared folder present on one of the machines.

Any recommendation?

Thanks

Answer 1

Multiplatform, locking. You best option is portalocker (https://pypi.python.org/pypi/portalocker/0.3). We've used it extensively -- and it's a winner !