In Python, how can I chain multiple processes together with popen, while redirecting intermediate partial results into a string

Question

I have been looking for a method in python to chain multiple processes together while redirecting stderr of some intermediate processes to a string variable.

After looking around, I think using subprocess.popen and subprocess.communicate provides the best solution. As an example code shown below:

import sys
import tempfile
from subprocess import Popen, PIPE


cmd = [sys.executable, '-c', 'print raw_input()']

# Using a temp file to give input data to the subprocess instead of stdin.write to avoid deadlocks.
with tempfile.TemporaryFile() as f:
    f.write('foobar')
    f.seek(0)  # Return at the start of the file so that the subprocess p1 can read what we wrote.
    p1 = Popen(cmd, stdin=f, stdout=PIPE)

p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE, stderr=PIPE)
p3 = Popen(cmd, stdin=p2.stdout, stdout=PIPE, stderr=PIPE)

# No order needed.
p1.stdout.close()
p2.stdout.close()

# Using communicate() instead of stdout.read to avoid deadlocks. 
print p3.communicate()[0]

Above code works perfectly fine if I'm just redirecting stdout of p2 to stdin of p3.

However, I also want to collect stderr output from p2 and process them later. Unfortunately, I cannot find a way to redirect p2.stderr into a variable (in this chained popen situation). I have tried using p2.stderr.read() or just get lines from p2.stderr but both failed. In particular, when the output of p2.stderr is large, it hangs p3 and the entire program goes into a deadlock.

Is there anyway to store p2.stderr into some variable (because I need to process the output later) without hanging up the program?

Answer 1

I had a similar problem, and this may not be ideal solution, but you can redirect the output to a text file and then read it from there:

tempFile = open("/tmp/myAppName.file.txt","w")
p2 = Popen(cmd, stdin=p1.stdout, stdout=PIPE, stderr=tempFile)
p2out = ""
#when you need the output, just do this:
with open("/tmp/myAppName.file.txt","r") as f:
     p2out = f.read()