How do I combine two files without overwriting any data in python?

Question

I need to concatenate two files, one which contains a single number and the other which contains at least two rows of data. I have tried shutil.copyfile(file2,file1) and subprocess.call("cat " + file2 + " >> " + file1, shell=True), both things give me the same result. The file with the single number contains an integer and a newline (i.e. two characters) so when I bring the two files together the first two characters of file2 are overwritten instead of just added to the end. If I do it through the shell using "cat file2 >> file1" this does not happen and it works perfectly.

Here is what I mean:

import numpy as np
from subprocess import call

f.open(file1)
f.write('2\n')
np.savetxt(file2,datafile,fmt)
call("cat " + file2 " >> " + file1, shell=True)

So instead of getting:

2
data data data ...
data data data ...

I get:

2
ta data data ...
data data data ...

I have no idea what is causing this issue but it is very frustrating. Any suggestions?

Answer 1

The problem is that you haven't flushed f. "2\n" is still in the file buffer and overwrites the other data after cat completes when f is eventually closed. But there Is a better way to do this. Reading the numpy docs savetxt, you can pass in a file handle. Numpy can use the existing file handle to write its data. No need for a second temporary file.

import numpy as np

with open(file1, "w") as f:
    f.write('2\n')
    np.savetxt(f, datafile, fmt)

Answer 2

Have you tried closing file1 first?

 f.close()
 np.savetxt... Etc

Answer 3

To append file2 to file1, you could use 'a' file mode as @krookik suggested:

from shutil import copyfileobj

with open(file1, 'w') as f: # write file1
    f.write('2\n')
# `with`-statement closes the file automatically

# append file2 to file1
with open(file2, 'rb') as input_file, open(file1, 'ab') as output_file:
    copyfileobj(input_file, output_file)

Your code doesn't work because it might be missing f.flush() or f.close() after f.write('2\n') as @beroe suggested i.e., when the cat command appends to file1, its content is not flushed from memory to disk yet, '2\n' is written later (when the file is closed implicitly on the program exit) and therefore it overwrites content written by cat.