error of consuming more than 7 GB memory even though no large data structure is used in Python on Win7

Question

I am working on python 2.7 to compare two txt files on win7.

I do not use any large data structure. But, it consume more than 7.78 GB memory. All files are located at local drives.

file1 = open(myfile1, 'r')
file2 = open(myfile2, 'r')
result_file1 = open(result_file, 'w')

while 1:
    l1 = file1.readline()
    l2 = file2.readline()
    if (l1 is EOFError or l2 is EOFError):
        break
    if (l1 != l2):
        a_line = l1.strip().split(',')
        b_line = l2.strip().split(',')
        if(a_line[0] != b_line[0] or a_line[1] != b_line[1]):
            print("error " + str(a_line) + " col not match " + str(b_line))
            return (1)     
        error = abs(float(a_line[2]) - float(b_line[2]))/float(a_line[2])
        result_file1.write(a_line[0] + "," + a_line[1]+ "," + a_line[2]
                    + "," + b_line[3] + "," + str(rel_error*100) + "\n")
  result_file1.close()
  file1.close()
  file2.close()

The result_file1 is not more than 50 MB.

thanks !

Answer 1

well, in your code there's no obvious reason why it's taking that much memory, though it's a better idea to use context managers to close the files when the function ends. As you iterate over the files until one of them fails, it's also a good idea to iterate both iterators at once. Also note that if you're using python3, you shall use zip() instead of izip(), as izip() is deprecated.

try:
    with open(myfile1, 'r') as file1:
        with open(myfile2, 'r') as file2:
            with open(result_file, 'w') as file3:
                for l1, l2 in izip(file1, file2):
                    if (l1 != l2):
                        a_line = l1.strip().split(',')
                        b_line = l2.strip().split(',')
                        if(a_line[0] != b_line[0] or a_line[1] != b_line[1]):
                            raise Exception("error {} could not match {}".format(a_line, b_line))
                        error = abs(float(a_line[2]) - float(b_line[2]))/float(a_line[2])
                        result_file1.write("{},{},{},{},{}\n".format(a_line[0],
                                                                     a_line[1],
                                                                     a_line[2],
                                                                     b_line[3],
                                                                     rel_error*100)
except Exception as err:
    print err

given the code I'm giving here, you should never have more than one line of each file loaded in memory at once. If you still get memory issues, it's either something else in your code that may be misbehaving, or your system not deallocating the memory fast enough.