Communicating with Jython/Java from Python (Subprocess)

Question

My problem is somewhat similar to this question.

I try to communicate from Python with a Jython program (that needs to keep running because it communicates with a Java API).

However, i can't get the output in realtime, whatever i try:

p2=Popen([path_jython],stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=False)
p2.stdin.write('test="hallo"\n')
p2.stdin.write('print test\n')
p2.stdin.flush()

for line in p2.stdout:
    print line

Nothing happens, the program blocks. It doesn't change when i iterate over p2.stdout.readlines() or iter(md5.stdout.readline, '') or when I repeatedly call p2.stdout.read(1) like suggested in the linked question.

If i add print p2.communicate() after the flush() however, i get the desired output:

>>> ('hallo\r\n', '\r\n')

but the program terminates afterwards...

Does anyone have a solution for this problem? Or are there alternatives to communicate effectively with a running Jython process? Happy for any advice!

EDIT: Python 2.7.5, Jython 2.7.1b

Answer 1

1. try to make jython stdout/stderr unbuffered, pass -u command-line argument
2. if you set stderr=PIPE then you should read it, otherwise a deadlock may occur if the child process fills the OS pipe buffer corresponding to its stderr. You could redirect it to stdout: stderr=STDOUT
3. set Python side buffering explicitely: bufsize=1 (line-buffered)
4. use iter(p.stdout.readline, '') to avoid read-ahead bug on Python 2

from subprocess import Popen, PIPE, STDOUT

p = Popen(['jython', '-u', ...], stdin=PIPE, stdout=PIPE, stderr=STDOUT,
          bufsize=1)
print >>p.stdin, 'test="hallo"' #NOTE: it uses `os.linesep`
print >>p.stdin, 'print test'
p.stdin.close() # + implicit .flush()
for line in iter(p.stdout.readline, ''):
    print line, #NOTE: comma -- softspace hack to avoid duplicate newlines
p.stdout.close()
rc = p.wait()