Python - capture Popen stdout AND display on console?

Question

I want to capture stdout from a long-ish running process started via subprocess.Popen(...) so I'm using stdout=PIPE as an arg.

However, because it's a long running process I also want to send the output to the console (as if I hadn't piped it) to give the user of the script an idea that it's still working.

Is this at all possible?

Cheers.

Answer 1

Can you simply print it as you read it from the pipe?

Answer 2

The buffering your long-running sub-process is probably performing will make your console output jerky and very bad UX. I suggest you consider instead using pexpect (or, on Windows, wexpect) to defeat such buffering and get smooth, regular output from the sub-process. For example (on just about any unix-y system, after installing pexpect):

>>> import pexpect
>>> child = pexpect.spawn('/bin/bash -c "echo ba; sleep 1; echo bu"', logfile=sys.stdout); x=child.expect(pexpect.EOF); child.close()
ba
bu
>>> child.before
'ba\r\nbu\r\n'

The ba and bu will come with the proper timing (about a second between them). Note the output is not subject to normal terminal processing, so the carriage returns are left in there -- you'll need to post-process the string yourself (just a simple .replace!-) if you need \n as end-of-line markers (the lack of processing is important just in case the sub-process is writing binary data to its stdout -- this ensures all the data's left intact!-).

Answer 3

S. Lott's comment points to Getting realtime output using subprocess and Real-time intercepting of stdout from another process in Python

I'm curious that Alex's answer here is different from his answer 1085071. My simple little experiments with the answers in the two other referenced questions has given good results...

I went and looked at wexpect as per Alex's answer above, but I have to say reading the comments in the code I was not left a very good feeling about using it.

I guess the meta-question here is when will pexpect/wexpect be one of the Included Batteries?

Answer 4

Inspired by pty.openpty() suggestion somewhere above, tested on python2.6, linux. Publishing since it took a while to make this working properly, w/o buffering...

def call_and_peek_output(cmd, shell=False):
    import pty, subprocess
    master, slave = pty.openpty()
    p = subprocess.Popen(cmd, shell=shell, stdin=None, stdout=slave, close_fds=True)
    os.close(slave)
    line = ""
    while True:
        try:
            ch = os.read(master, 1)
        except OSError:
            # We get this exception when the spawn process closes all references to the
            # pty descriptor which we passed him to use for stdout
            # (typically when it and its childs exit)
            break
        line += ch
        sys.stdout.write(ch)
        if ch == '\n':
            yield line
            line = ""
    if line:
        yield line

    ret = p.wait()
    if ret:
        raise subprocess.CalledProcessError(ret, cmd)

for l in call_and_peek_output("ls /", shell=True):
    pass

Answer 5

Alternatively, you can pipe your process into tee and capture only one of the streams. Something along the lines of sh -c 'process interesting stuff' | tee /dev/stderr.

Of course, this only works on Unix-like systems.