Python: clear stdin Buffer

Question

This is possibly a nooby question but i cant find a way to do it! I need to clear the stdin buffer in python.

Imagine i have following bash script running:

i=0
for (( ; ; ))
do
    echo "$i"
    ((i++))
done

run from the command line like so: ./loop.sh | python myProg.py

in myProg.py i wish to have something as follows:

count = 100
f = fileinput.input()
while True:
    sleep(2)
    # clear stdin somehow ...
    # and read the most recent 100 lines
    i = 0
    while i < count:
        myBuffer[i] = f.readline()
        if len(myBuffer[i]) > 0:
            i += 1
    print myBuffer

i don't think i can just read all the lines, before it because it is spitting them out at a high rate and if sleep (just for testing atm) is a few minutes it seems silly ... is there a way to set the stdin buffer size in python? Or just truncate/clear it? Btw im using python 2 so no bufsize argument

I had a look here How to avoid Python fileinput buffering

Any python way of doing it? But ill try unbuffer too: https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Update: no luck with unbuffer or stdbuf ...

Answer 1

Just in case anyone else has this issue i have been on an extensive search and come to the conclusion it is not possible due to kernel implementation etc ...

I wrote the following work around to do what i wanted. I created two files: textFilter.py and getLatest.py. Essentially you run ./loopPrinting.sh | python textFilter.py and get the Latest 100 lines. I am reasonably sure this is atomic and reliable (though if it isn't please tell me!!!). It creates 16 files and changes the count (similar to the apple live streaming i think).

textFilter.py

import sys
import os
import time

def mainLoop(size):
    myBuffer = [0]*size
    count = 0

    while True:
        for i in range(size):
            myBuffer[i] = sys.stdin.readline()

        f = open('/home/development/textFilter/' + repr(count) + '.dat', 'w')
        f.write(''.join(myBuffer))
        f.close()

        f = open('/home/development/textFilter/count.dat~', 'w')
        f.write(repr(count))
        f.flush()
        os.fsync(f.fileno()) 
        f.close()
        time.sleep(0.01)
        os.rename('/home/development/textFilter/count.dat~','/home/development/textFilter/count.dat')

        count += 1
        count = count % 16

if __name__ == "__main__":
    try:
        mainLoop(int(sys.argv[1]))
    except Exception:
        mainLoop(100)

getLatest.py

import sys
import time

def getData():
    f = open('count.dat', 'r')
    count = f.read()
    f.close()

    count = int(count)
    oldCount = count

    while oldCount == count:
    time.sleep(0.1)
    f = open('count.dat', 'r')
    count = f.read()
    f.close()
    count = int(count)

    f = open(repr(count) + '.dat', 'r')
    data = f.readlines()
    f.close()

    for row in data:
    print row,

    return 0

if __name__ == "__main__":
    try:
    getData()
    except Exception as inst:
    sys.stderr.write('ERROR')
    sys.stderr.write(inst)