Trouble printing text live with Python subprocess.call

Question

Off the bat, here is what I am importing: import os, shutil from subprocess import call, PIPE, STDOUT

I have a line of code that calls bjam to compile a library:

call(['./bjam',
      '-j8',
      '--prefix="' + tools_dir + '"'],
     stdout=PIPE)

I want it to print out text as the compilation occurs. Instead, it prints everything out at the end.

It does not print anything when I run it like this. I have tried running the command outside of Python and determined that all of the output is to stdout (when I did ./bjam -j8 > /dev/null I got no output, and when I ran ./bjam -j8 2> /dev/null I got output).

What am I doing wrong here? I want to print the output from call live.

---

As a sidenote, I also noticed something when I was outputting the results of a git clone operation:

call(['git',
      'clone', 'https://github.com/moses-smt/mosesdecoder.git'],
     stdout=PIPE)

prints the stdout text live as the call process is run.

call(['git',
      'clone', 'https://github.com/moses-smt/mosesdecoder.git'],
     stdout=PIPE, stderr=STDOUT)

does not print out any text. What is going on here?

Answer 1

Try:

def run(command):
    proc = subprocess.Popen(command, stdout=subprocess.PIPE)
    for lineno, line in enumerate(proc.stdout):
        try:
            print(line.decode('utf-8').replace('\n', ''))
        except UnicodeDecodeError:
            print('error(%d): cannot decode %s' % (lineno, line))

The try...except logic is for python 3 (maybe 3.2/3.3, I'm not sure), as there line is a byte array not a string. For earlier versions of python, you should be able to do:

def run(command):
    proc = subprocess.Popen(command, stdout=subprocess.PIPE)
    for line in proc.stdout:
        print(line.replace('\n', ''))

Now, you can do:

run(['./bjam', '-j8', '--prefix="' + tools_dir + '"'])

Answer 2

stdout=PIPE redirects subprocess' stdout to a pipe. Don't do it unless you want to read from the subprocesses stdout in your code using proc.communicate() method or using proc.stdout attribute directly.

If you remove it then subprocess should print to stdout like it does in the shell:

from subprocess import check_call

check_call(['./bjam', '-j8', '--prefix', tools_dir])

I've used check_call() to raise an exception if the child process fails.

See Python: read streaming input from subprocess.communicate() if you want to read subprocess' output line by line (making the line available as a variable in Python) as soon as it is avaiable.

Answer 3

call will not print anything it captures. As documentation says "Do not use stdout=PIPE or stderr=PIPE with this function. As the pipes are not being read in the current process, the child process may block if it generates enough output to a pipe to fill up the OS pipe buffer."

Consider using check_output and print its return value.

In the first case with git call you are not capturing stderr and therefor it normally flows onto your terminal.