Redirecting stdout from a python script called in *nix environment does not work correctly

Question

I have a Python program in which I have some Print statements, followed by calls to some c++ executables from which I also get some output to the stdout. When I run this python script in a unix terminal I get output on the screen as expected (in the correct order i.e. first from the print an then from c++ executables). The problem is when I redirect this output to a file like

python test.py > out.txt

I get the output in the wrong order. I get the output of the c++ executables first and then the other.

Answer 1

Python uses line-buffering when stdout is a tty device, hence the output is as per print statements order. In case of redirection python buffers the output of print statements. While c++ executable output buffering is not handled by python in both cases. So when the output is redirected, the print statements are getting buffered and doesn't output to file till the buffer is full or program ends.

sys.stdout.flush() will flush the output of print statements as below. Note it should follow the print statements

#!/usr/local/bin/python
import os
import sys

print("Hello1")
print("Hello2")
sys.stdout.flush()
os.system("/python/cppprog.o")
print("Bye1")
print("Bye2")

Output:

]# python script.py > o.txt
]# cat o.txt
Hello1
Hello2
Hello, world!
Bye1
Bye2

Answer 2

You can run python with unbuffered output using the command line switch -u, i.e. you can just call python -u myscript.py and the output to stdout should be synchronized thereafter.

Answer 3

This is because python's stdout buffer and os' stdout buffer are not synced.

Try flushing stdout after your print statements and before executing c++ executables.

import sys
sys.stdout.flush()