Output from py2exe converted program gets received all at once

Question

I have made a python console script, and converted it to a .exe with py2exe (I have used the console = ['test.py'] line in my setup file).

The program parses a file, and during the parse it prints out how much of the file it has parsed. Typical output would be:

Processing (currently at 1%)
Processing (currently at 4%)

etc. When running the file in a cmd window it works just as expected.

I have also created a very small C# WPF program that just runs the parser:

Process p = new Process();
p.StartInfo = new ProcessStartInfo(@"C:\temp\test.exe");
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.UseShellExecute = false;
p.StartInfo.CreateNoWindow = true;

p.OutputDataReceived += new DataReceivedEventHandler(p_OutputDataReceived);

Task t = new Task(() =>
{
    p.Start();
    p.BeginOutputReadLine();
    p.WaitForExit();
    p.Close();
});
t.Start();

The p_OutputDataReceived handler just sends the received output to a textbox. This works and I have tested it on other programs, and there I get the output from the program when I expect.

However when I run my parser (the one created with py2exe) I get all outputs just after the parser has finished. So in the end I get all the correct output, but I get them all at the same time...

(note, I don't get one big output, but rather all the expected outputs but still, all at the same time)

---

So to be perfectly clear here:

• If I run the parser from the command window, I get the outputs one by one
• I have tested to run a C# console program instead of the py2exe generated program and that works (I get the outputs one by one)

Answer 1

Python checks if sys.stdout (the program's standard output) is a console. If it is, Python flushes write buffers immediately so the user can see it. Otherwise writes get cached and are outputted all at once:

• when the write buffer is full or
• at program exit.

The logic behind this is better performance, since when redirecting stdout to a file or to other programs, typically no one cares when the output occurs.

You can fix this by including sys.stdout.flush() in strategic locations in your Python parser script (i.e. directly after printing status line).

Btw: You should be able to observe the same time-delay behavior if you redirect the output of your parser to more, for example:

C:\temp\test.exe | more