with open inside try - except block, too many files open?

Question

Quite simply, I am cycling through all sub folders in a specific location, and collecting a few numbers from three different files.

def GrepData():
import glob as glob
import os as os

os.chdir('RUNS')
RUNSDir = os.getcwd()
Directories = glob.glob('*.*')
ObjVal = []
ParVal = []
AADVal = []

for dir in Directories:
    os.chdir(dir)
    (X,Y) = dir.split(sep='+')
    AADPath = glob.glob('Aad.out')
    ObjPath = glob.glob('fobj.out')
    ParPath = glob.glob('Par.out')

    try:
        with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
            for line in ObjFile:
                ObjVal.append(list([X,Y,line.split()[0]]))
        ObjFile.close()
    except(IndexError):
        ObjFile.close()

    try:
        with open(os.path.join(os.getcwd(),ParPath[0])) as ParFile:
            for line in ParFile:
                ParVal.append(list([X,Y,line.split()[0]]))
        ParFile.close()
    except(IndexError):
        ParFile.close()
    try:
        with open(os.path.join(os.getcwd(),AADPath[0])) as AADFile:
            for line in AADFile:
                AADVal.append(list([X,Y,line.split()[0]]))
        AADFile.close()
    except(IndexError):
        AADFile.close()
    os.chdir(RUNSDir)

Each file open command is placed in a try - except block, as in a few cases the file that is opened will be empty, and thus appending the line.split() will lead to an index error since the list is empty.

However when running this script i get the following error: "OSError: [Errno 24] Too Many open files"

I was under the impression that the idea of the "with open..." statement was that it took care of closing the file after use? Clearly that is not happening.

So what I am asking for is two things:

1. The answer to: "Is my understanding of with open correct?"
2. How can I correct whatever error is inducing this problem?

(And yes i know the code is not exactly elegant. The whole try - except ought to be a single object that is reused - but I will fix that after figuring out this error)

Answer 1

Try moving your try-except inside the with like so:

with open(os.path.join(os.getcwd(),ObjPath[0])) as ObjFile:
    for line in ObjFile:
        try:
            ObjVal.append(list([X,Y,line.split()[0]]))
        except(IndexError): 
           pass

Notes: there is no need to close your file manually, this is what with is for. Also, there is no need to use as os in your imports if you are using the same name.

Answer 2

"Too many open files" has nothing to do with writing semantically incorrect python code, and you are using with correctly. The key is the part of your error that says "OSError," which refers to the underlying operating system.

When you call open(), the python interpreter will execute a system call. The details of the system call vary a bit by which OS you are using, but on linux this call is open(2). The operating system kernel will handle the system call. While the file is open, it has an entry in the system file table and takes up OS resources -- this means effectively it is "taking up space" whilst it is open. As such the OS has a limit to the number of files that can be opened at any one time.

Your problem is that while you call open(), you don't call close() quickly enough. In the event that your directory structure requires you to have many thousands files open at once that might approach this cap, it can be temporarily changed (at least on linux, I'm less familiar with other OSes so I don't want to go into too many details about how to do this across platforms).