Need to run a diff command on 2 NamedTemporaryFiles using subprocess module

Question

I am trying to run a diff on 2 named temporary files, I did not use difflib because its output was different from the linux diff. When I run this code, It does not output anything. I tried a diff on regular files and that works just fine.

#using python 2.6
temp_stage = tempfile.NamedTemporaryFile(delete = False)
temp_prod = tempfile.NamedTemporaryFile(delete = False)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)

#this does not work, shows no output, tried both call and popen
subprocess.Popen(["diff", temp_stage.name, temp_prod.name])

#subprocess.call(["diff", temp_stage.name, temp_prod.name])

Answer 1

You need to force the files to be written out to disk by calling flush(); or else the data you were writing to the file may only exist in a buffer.

In fact, if you do this, you can even use delete = True, assuming there's no other reason to keep the files around. This keeps the benefit of using tempfile.

#!/usr/bin/python2
temp_stage = tempfile.NamedTemporaryFile(delete = True)
temp_prod = tempfile.NamedTemporaryFile(delete = True)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)

temp_stage.flush()
temp_prod.flush()

subprocess.Popen(["diff", temp_stage.name, temp_prod.name])

Answer 2

Unrelated to your .flush() issue, you could pass one file via stdin instead of writing data to disk:

from tempfile import NamedTemporaryFile
from subprocess import Popen, PIPE

with NamedTemporaryFile() as file:
    file.write(prod_notes)
    file.flush()
    p = Popen(['diff', '-', file.name], stdin=PIPE)
    p.communicate(stage_notes) # diff reads the first file from stdin

if p.returncode == 0:
    print('the same')
elif p.returncode == 1:
    print('different')
else:
    print('error %s' % p.returncode)

diff reads from stdin if input filename is -.

If you use a named pipe then you don't need to write data to disk at all:

from subprocess import Popen, PIPE
from threading import Thread

with named_pipe() as path:
    p = Popen(['diff', '-', path], stdin=PIPE)
    # use thread, to support content larger than the pipe buffer
    Thread(target=p.communicate, args=[stage_notes]).start()
    with open(path, 'wb') as pipe:
        pipe.write(prod_notes)

if p.wait() == 0:
    print('the same')
elif p.returncode == 1:
    print('different')
else:
    print('error %s' % p.returncode)

where named_pipe() context manager is defined as:

import os
import tempfile
from contextlib import contextmanager
from shutil import rmtree

@contextmanager
def named_pipe(name='named_pipe'):
    dirname = tempfile.mkdtemp()
    try:
        path = os.path.join(dirname, name)
        os.mkfifo(path)
        yield path
    finally:
        rmtree(dirname)

The content of a named pipe doesn't touch the disk.

Answer 3

I would suggest bypassing the tempfile handling since with a NTF you're going to have to handle cleanup anyway. Create a new file and write your data then close it. Flush the buffer and then call the subprocess commands. See if that gets it to run.

f=open('file1.blah','w')
f2=open('file2.blah','w')
f.write(stage_notes)
f.flush()
f.close()
f2.write(prod_notes)
f2.flush()
f2.close()

then run your subprocess calls