File copying in python stops after 50000 lines

Question

I have been trying to copy a file to another using Python using the following code.

To avoid confusion, please note that the snippet given is inside a loop which carries out a loop on aij[j].

import glob
import shutil 
for filename in glob.glob("dpd.*.txt"):

    print "File under process ", filename

    filedummy = filename + '_' + str(aij[j])

    with open(filename,'r') as infyle, open(filedummy,'w') as outfyle:

        for line in infyle:

            outfyle.write(line)

        srcfile = filedummy
        destfile = path_back + '/' + filedummy
        shutil.copy(srcfile,destfile)

Problem : Number of lines in the original file is about 90000 lines. Upto 50026 lines, there is no problem in copying after which the copying abruptly stops. Any help will be appreciated.

Answer 1

The problem here is that you're copying the temporary file before it has been closed.

The file is still open, so the shutil.copy is reading whatever is on disk at that point, which is not the entire file. Some of it is still held in buffers. These buffers will be flushed to the disk when you close the file.

So just move the last 3 lines of your code in the question out one level to this:

with open(filename,'r') as infyle, open(filedummy,'w') as outfyle:
    for line in infyle:
        outfyle.write(line)

srcfile = filedummy
destfile = path_back + '/' + filedummy
shutil.copy(srcfile,destfile)

This will close the file, flushing the buffers, and then you can copy it.