Python (2.7) script monitoring and notification system

Question

I've read a lot of other posts about monitoring python scripts, but haven't been able to find anything like what I am hoping to do. Essentially, I have 2 desktops running Linux. Each computer has multiple python scripts running non-stop 24/7. Most of them are web scraping, while a few others are scrubbing and processing data. I have built pretty extensive exception handling into them that sends me an email in the event of any error or crash, but there are some situations that I haven't been able to get emailed about (such as if the script itself just freezes or the computer itself crashes, or the computer looses internet connection, etc.)

So, I'm trying to build a sort of check-in service where a python script checks in to the service multiple times throughout it's run, and if it doesn't check-in within X amount of time, then send me an email. I don't know if this is something that can be done with the signal or asyncore module(s) and/or sockets, or what a good place would be to even start.

Has anyone had any experience in writing anything like this? Or can point me in the right direction?

Answer 1

Take a look at supervision tools like monit or supervisord. Those tools are built to do what you described. For example: create a simple init.d script for your python process:

PID_FILE=/var/run/myscript.pid                                               
LOG_FILE=/mnt/logs/myscript.log                                                  
SOURCE=/usr/local/src/myscript                                           

case $1 in                                                                      
  start)                                                                        
    exec /usr/bin/python $SOURCE/main_tread.py >> LOG_FILE 2>&1 &                     
    echo $! > $PID_FILE                                                         
    ;;                                                                          
  stop)                                                                         
    kill `cat ${PID_FILE}`                                                      
    ;;                                                                          
  *)                                                                            
    echo "Usage: wrapper {start|stop}"                                          
    ;;                                                                          

esac                                                                            
exit 0  

Then add this to the monit config:

check process myscript pidfile /var/run/myscript.pid
  start program = "/etc/init.d/myscript start"
  stop program = "/etc/init.d/myscript stop"

check file myscript.pid path /var/run/myscript.pid
  if changed checksum then alert

Also check documentation, it has pretty good example on how to setup alerts and send emails.

Answer 2

Upstart is a good choice but I' afraid it is only available for Ubuntu and Redhat based distros