how to use pexpect to get spontaneous output of subprocess in python

Question

This is related to my another post multithreading issue with wx.TextCtrl (or underlying GTK+), which after correction with calling GUI interactions from primary thread, I find it again comes to the pipe block buffering problem. so How to get spontaneous output from the subprocess.stdout?

To be in short, currently I am using subprocess.popen to launch an external long-time running program.

    launchcmd=["EXTERNAL_PROGRAM_EXE"]
    p = subprocess.Popen(launchcmd, stdin=subprocess.PIPE, 
            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    self.outputThread = BashProcessThread(p.stdout.readline)
    self.outputThread.start()
    # wx.TextCtrl is used to make input/output
    self.textctrl = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)

And I use a separate thread to read the stdout of the background program, with "wx.CallAfter" to call back.

class BashProcessThread(threading.Thread):
    def __init__(self, readlineFunc, textctrl):
        threading.Thread.__init__(self)
        self.readlineFunc = readlineFunc

    def run(self):
        while True:
           line = self.readlineFunc()
           wx.CallAfter(textctrl.AppendText(line))

The above code prints out the subprocess log messages block-hanging-block (instead of spontaneously line by line), and the worst is the remaining 5-6 lines of log messages could not be timely printed until the user send the next input.

From my old post, I get to know there is pty and pexpect, which could make the subprocess thought it is interacting with pseudo-tty. But how should pexpect be used, especially considering the background process is long-term, independent running task?

e.g., If I used

child=pexpect.spawn(launchcmd)

How can I get the output and input of the subprocess, so I could use wx.TextCtrl to print the output, and also use wx.TextCtrl to forward user input to subprocess?

Answer 1

Have you tried something like:

child = pexpect.spawn(launchcmd)
while True:
    try:
        child.expect('\n')
        print(child.before)
    except pexpect.EOF:
        break

Answer 2

I found that these two methods work well for getting live output.

If you don't want the option for user interaction, like in a background process:

child = pexpect.spawn(launchcmd)
child.logfile = sys.stdout
child.expect(pexpect.EOF)
child.close()

If you weren't using a background process and want the ability to interact with the program (if it prompts you). What happens here is that you go into interactive mode and pexpect writes directly to the screen. When the program hits it's end/EOF it throws an OSError.

child = pexpect.spawn(launchcmd)
try:
    child.interact()
except OSError:
    pass
child.close()