Reading updated files on the fly in Python

Question

I'm writing two Python scripts that both parse files. One is a standard unix logfile and the other is a binary file. I'm trying to figure out the best way to monitor these so I can read data as soon as they're updated. Most of the solutions I've found thus far are linux specific, but I need this to work in FreeBSD.

Obviously one approach would be to just run my script every X amount of time, but this seems gross and inefficient. If I want my Python app running continuously in the background monitoring a file and acting on it once it's changed/updated, what's my best bet?

Answer 1

Have you tried KQueue events?

http://docs.python.org/library/select.html#kqueue-objects

kqueue is the FreeBSD / OS version of inotify (file change notification service). I haven't used this, but I think it's what you want.

Answer 2

I once did to make a sort of daemon process for a parser built in Python. I needed to watch a series of files and process them in Python, and it had to be a truly multi-OS solution (Windows & Linux in this case). I wrote a program that watches over a list of files by checking their modification time. The program sleeps for a while and then checks the modification time of the files being watched. If the modification time is newer than the one previously registered, then the file has changed and so stuff has to be done with this file.

Something like this:

import os
import time

path = os.path.dirname(__file__)
print "Looking for files in", path, "..."

# get interesting files
files = [{"file" : f} for f in os.listdir(path) if os.path.isfile(f) and os.path.splitext(f)[1].lower() == ".src"]
for f in files:
    f["output"] = os.path.splitext(f["file"])[0] + ".out"
    f["modtime"] = os.path.getmtime(f["file"]) - 10
    print "  watching:", f["file"]


while True:
    # sleep for a while
    time.sleep(0.5)
    # check if anything changed
    for f in files:
        # is mod time of file is newer than the one registered?
        if os.path.getmtime(f["file"]) > f["modtime"]: 
            # store new time and...
            f["modtime"] = os.path.getmtime(f["file"])
            print f["file"], "has changed..."
            # do your stuff here

It does not look like top notch code, but it works quite well.

There are other SO questions about this, but I don't know if they'll provide a direct answer to your question:

How to implement a pythonic equivalent of tail -F?

How do I watch a file for changes?

How can I "watch" a file for modification / change?

Hope this helps!