How to get output from external command combine with Pipe

Question

I have command like this.

wmctrl -lp | awk '/gedit/ { print $1 }'

And I want its output within python script, i tried this code

>>> import subprocess
>>> proc =  subprocess.Popen(["wmctrl -lp", "|","awk '/gedit/ {print $1}"], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> proc.stdout.readline()
'0x0160001b -1 6504   beer-laptop x-nautilus-desktop\n'
>>> proc.stdout.readline()
'0x0352f117  0 6963   beer-laptop How to get output from external command combine with Pipe - Stack Overflow - Chromium\n'
>>> proc.stdout.readline()
'0x01400003 -1 6503   beer-laptop Bottom Expanded Edge Panel\n'
>>> 

It seem my code is wrong only wmctrl -lp was execute, and | awk '{print $1}' is omitted My expect output would like 0x03800081

$ wmctrl -lp | awk '/gedit/ {print $1}'
0x03800081

Does one please help.

Answer 1

With shell=True, you should use a single command line instead of an array, otherwise your additional arguments are interpreted as shell arguments. From the subprocess documentation:

On Unix, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments.

So your call should be:

subprocess.Popen("wmctrl -lp | sed /gedit/ '{print $1}'", shell=True, ...

I think you may also have an unbalanced single quote in there.

Answer 2

Because you are passing a sequence in for the program, it thinks that the pipe is an argument to wmcrtrl, such as if you did

wmctrl -lp "|"

and thus the actual pipe operation is lost.

Making it a single string should indeed give you the correct result:

>>> import subprocess as s
>>> proc = s.Popen("echo hello | grep e", shell=True, stdout=s.PIPE, stderr=s.PIPE)
>>> proc.stdout.readline()
'hello\n'
>>> proc.stdout.readline()
''

Answer 3

After some research, I have the following code which works very well for me. It basically prints both stdout and stderr in real time. Hope it helps someone else who needs it.

stdout_result = 1
stderr_result = 1


def stdout_thread(pipe):
    global stdout_result
    while True:
        out = pipe.stdout.read(1)
        stdout_result = pipe.poll()
        if out == '' and stdout_result is not None:
            break

        if out != '':
            sys.stdout.write(out)
            sys.stdout.flush()


def stderr_thread(pipe):
    global stderr_result
    while True:
        err = pipe.stderr.read(1)
        stderr_result = pipe.poll()
        if err == '' and stderr_result is not None:
            break

        if err != '':
            sys.stdout.write(err)
            sys.stdout.flush()


def exec_command(command, cwd=None):
    if cwd is not None:
        print '[' + ' '.join(command) + '] in ' + cwd
    else:
        print '[' + ' '.join(command) + ']'

    p = subprocess.Popen(
        command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd
    )

    out_thread = threading.Thread(name='stdout_thread', target=stdout_thread, args=(p,))
    err_thread = threading.Thread(name='stderr_thread', target=stderr_thread, args=(p,))

    err_thread.start()
    out_thread.start()

    out_thread.join()
    err_thread.join()

    return stdout_result + stderr_result

When needed, I think it's easy to collect the output or error in a string and return.