Log tailing daemon with Python

Question

We have a legacy C written program that output log to a text file and this program is able to rotate the log daily.

e.g.

app.log
app.log.2012-10-24
app.log.2012-10-23
..

Now I want to write a python daemon that tail the log and inject into DB, my questions

1. What is the recommended approach to tail a file in Python? Twisted or Pyinotify?
2. How to handle the case the daemon was stopped and we need to resume the log tailing? Should I store the offset in another file?

Thanks.

Answer 1

I have encountered similar problems. My approach is this:

import io
import time

def checklog(fs,logfile):
    trytimes = 10
    line = ''
    while line == '':
        time.sleep(1)
        line = fs.readline().replace('\n','')
        trytimes = trytimes - 1
        while trytimes == 0 and line == '':
            try:
                fs = io.open(logfile,'rb')
                fs.seek(0,2)
                trytimes = 10
                break
            except:
                time.sleep(10)
    return line,fs


logfile="app.log"
fs = io.open(logfile,'rb')
fs.seek(0,2)# seek to tail of the logfile
while True:
    line = fs.readline().replace('\n','')
    while line == '':
        line,fs = checklog(fs,logfile)
        #do something for line;

Hope this helps.