Python dictionary too full or something else?

Question

I have about 1300 h5 files to sort into a dictionary using h5py. When I run my code on about 250 files it works fine, but any amount larger gives me the error:

  Traceback (most recent call last):
    File "solution_script.py", line 31, in <module>
    File "build/bdist.macosx-10.7-intel/egg/h5py/_hl/files.py", line 165, in __init__
    File "build/bdist.macosx-10.7-intel/egg/h5py/_hl/files.py", line 57, in make_fid
    File "h5f.pyx", line 70, in h5py.h5f.open (h5py/h5f.c:1626)
  IOError: unable to open file (File accessability: Unable to open file)'

I'm not sure if I set up the dictionary incorrectly or if there is a better way to do this.

Here's my code if anyone can see some obvious mistakes I've made:

nodesDictionary = {}
velDictionary = {}
cellsDictionary={}
num_h5file = -1;
for root, directory, files in os.walk(rootDirectory):
    for file in sorted(files):
        if file.endswith(".h5"):
            num_h5file = num_h5file + 1
                curh5file = h5py.File(file, 'r')
                nodes_list = curh5file["nodes"]
                cells_list = curh5file["cells"]
                velocity_list = curh5file["velocity"]
                for i in range(0, len(nodes_list)-1):
                    if num_h5file not in nodesDictionary:
                            nodesDictionary[num_h5file] = [curh5file["nodes"][i]]
                            velDictionary[num_h5file] = [curh5file["velocity"][i]]                         
                    else:
                            nodesDictionary[num_h5file].append(curh5file["nodes"][i])
                            velDictionary[num_h5file].append(curh5file["velocity"][i])
                for j in range(0, len(cells_list)-1):
                    if num_h5file not in cellsDictionary:
                            cellsDictionary[num_h5file] = [curh5file["cells"][j]]
                    else:
                            cellsDictionary[num_h5file].append(curh5file["cells"][j])

Any help/advice/insight would be greatly appreciated :)

Answer 1

I agree with @reptilicus that your best bet is to close the files you aren't actively using if possible. But if you really must, you can use the ulimit command to increase the number of open files available to your process - see this answer for details

Edit:

You can use context management to ensure the file is closed even if an error occurs:

with h5py.File(file, 'r') as curh5file:
    ... # Do your stuff

... # continue other actions

When you exit the "with" block, the file will be automatically closed. If an exception happens, it will still be closed.

Answer 2

You probably need to close the files when you are done working on them, there is a limit in the OS on how many files you can open at a time.

At the end of your block where you open the h5 file, just put in a close statement and all should be well. Something like:

curh5file = h5py.File(file, 'r')
...do some stuff
curh5file.close()