How to concatenate the output of several processes into the input of another?

Question

I'm writing a script that executes a list of processes and concatenates all of their output into the input of another process. I've condensed my script into a test case using echo and cat as stand-ins for the actual processes.

#!/usr/bin/python

import os,subprocess

(pipeOut, pipeIn) = os.pipe()

catProcess = subprocess.Popen("/bin/cat", stdin = pipeOut)

for line in ["First line", "Last line"]:
    subprocess.call(["/bin/echo",line], stdout = pipeIn)

os.close(pipeIn)
os.close(pipeOut)

catProcess.wait()

The program works as expected, except that the call to catProcess.wait() hangs (presumably because it's still waiting for more input). Passing close_fds=True to Popen or call doesn't seem to help, either.

Is there a way to close catProcesses's stdin so it exits gracefully? Or is there another way to write this program?

Answer 1

Passing close_fds=True to catProcess helps on my system.

You don't need to create the pipe explicitly:

#!/usr/bin/python
from subprocess import Popen, PIPE, call

cat = Popen("cat", stdin=PIPE)
for line in ["First line", "Last line"]:
    call(["echo", line], stdout=cat.stdin)
cat.communicate() # close stdin, wait