Call Perl script from Python constantly returning values

Question

I found a question on this site which showed me how to call a Perl script from Python. I'm currently using the following lines of code to achieve this:

pipe = subprocess.Popen(["perl", "./Perl_Script.pl", param], stdout=subprocess.PIPE)
result = pipe.stdout.read()

This works perfectly, but the only issue is that the Perl script takes a few minutes to run. At the end of the Perl script, I use a simple print statement to print my values I need to return back to Python, which gets set to the result variable in Python.

Is there a way I can include more print statements in my Perl script every few seconds that can get returned to Python continuously (instead of waiting a few minutes and returning a long list at the end)?

Ultimately, what I'm doing is using the Perl script to obtain data points that I then send back to Python to plot an eye diagram. Instead of waiting for minutes to plot the eye diagram when the Perl script is finished running, I'd like to return segments of the data to Python continuously, allowing my plot to update every few seconds.

Answer 1

You need two pieces: To read a line at a time in Python space and to emit a line at a time from Perl. The first can be accomplished with a loop like

while True:
  result = pipe.stdout.readline()
  if not result:
    break
  # do something with result

The readline blocks until a line of text (or EOF) is received from the attached process, then gives you the data it read. So long as each chunk of data is on its own line, that should work.

If you run this code without modifying the Perl script, however, you will not get any output for quite a while, possibly until the Perl script is finished executing. This is because Perl block-buffers output to a pipe by default. You can tell it to flush the buffer more often by changing a global variable in the scope in which you are printing:

use English qw(-no_match_vars);
local $OUTPUT_AUTOFLUSH = 1;
print ...;

See http://perl.plover.com/FAQs/Buffering.html and http://perldoc.perl.org/perlvar.html .

Answer 2

The default UNIX stdio buffer is at least 8k. If you're writing less than 8k, you'll end up waiting until the program ends before the buffer is flushed.

Tell the Perl program to stop buffering output, and probably tell python not to buffer input through the pipe.

$| = 1;

to un-buffer STDOUT in your Perl program.

Answer 3

pipe.stdout.read() tries to read the whole stream, so it will block until perl is finished. Try this:

line=' '
while line:
    line = pipe.stdout.readline()
    print line,