Work with the stdout of some other process (created with Popen)

Question

I would like to count the number of lines written to stdout by a process (here unrar.exe) created with Popen.

import time
from subprocess import Popen, PIPE, STDOUT

p = Popen('unrar.exe x -y myfile.rar', stdout=PIPE)

while (p is not finished):      # pseudo code here and next lines...
    time.sleep(0.100)
    print 'Number of lines written to STDOUT by unrar' + len(PIPE.split('\n'))

How to do this properly ?

Remark : I already looked at p.communicate() (https://python.readthedocs.org/en/v2.7.2/library/subprocess.html#subprocess.Popen.communicate) but this has the effect of blocking the execution of the Python until p has terminated, which is not what I want : I want to be able to print the number of lines written by p when it's running.

Answer 1

I'm looking for an easier solution without using another thread, etc. I don't mind if the process is blocked every 100 ms

If it is a hard requirement that the process must not block then you need threads (or other asynchronous techniques). To emulate non-blocking wc --lines <(cmd):

#!/usr/bin/env python
import io
import shlex
from functools import partial
from subprocess import Popen, PIPE
from threading import Thread
from Queue import Queue

def count_lines(pipe, queue, chunksize=io.DEFAULT_BUFFER_SIZE):
    #NOTE: you could write intermediate results here (just drop `sum()`)
    queue.put(sum(chunk.count(b'\n')
                  for chunk in iter(partial(pipe.read, chunksize), b'')))
    pipe.close()

p = Popen(shlex.split('unrar.exe x -y myfile.rar'), stdout=PIPE, bufsize=-1)
result = Queue()
Thread(target=count_lines, args=[p.stdout, result]).start()
p.wait() # you can omit it if you want to do something else and call it later
number_of_lines = result.get() # this blocks (you could pass `timeout`)

---

On the other hand if all you need is "to print the number of lines written by p when it's running." then you could count lines in the main thread:

#!/usr/bin/env python
import shlex
from subprocess import Popen, PIPE

p = Popen(shlex.split('unrar.exe x -y myfile.rar'), stdout=PIPE, bufsize=1)
count = 0
for line in iter(p.stdout.readline, b''):
    count += 1
    print count
p.stdout.close()
p.wait()

Answer 2

I don't know if this is the cleanest way to do it, but it works:

from subprocess import Popen, PIPE
import io, time

p = Popen('unrar.exe x -y myfile.rar', stdout = PIPE)

b = ' '
i = 0 
while b:
    b = p.stdout.readline()
    i += 1
    print i

print 'FINISHED'