Monitoring for Program Crash

Question

I'm currently working with the Python module Spynner for automating some web tasks. I've run in to a problem though where for some reason the process would just simply stop moving, freeze but still be responsive according to Windows.

What I'd like to do is setup some form of a monitor to check and see if this happens, and then restart the process. I was thinking of possibly monitoring the terminal output of the program, and if it stops pushing data after a certain amount of time, it would kill the program and start up again.

I know how I'd like to kill the program and run it again, simply using os and subprocess, but I'm not sure how to setup the piece to monitor if the terminal stops sending data for a specific amount of time.

Answer 1

The following code is borrowed and slightly modified from "Non-blocking read on a subprocess.PIPE in python" (kudos to J.F. Sebastian - if you accept this answer, please upvote the original code)

import sys
import time
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        timestamp = time.time( )
        queue.put((timestamp, line))
    out.close()

#-- This is how long you're willing to wait before you 
#-- consider your Spynner process to be brain-dead.
MAX_WAIT_TIME = 300.0  #-- we'll wait 5 minutes (300 seconds)

#-- Construct a shared queue that will be used to send messages from 
#-- the subprocess I/O polling thread to the watchdog (main) thread.
q = Queue()

#-- Spawn your subprocess...
p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)

#-- Create a new thread that runs in the same process as the watchdog.
#-- This thread will poll the output of the subprocess and populate the 
#-- shared queue.
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()

#-- Now, we'll try to read from the shared queue.
try:
    #-- Queries the shared queue for the next item in the queue,
    #-- waiting for up to MAX_WAIT_TIME before failing with an Empty exception.
    timestamp, line = q.get(True, MAX_WAIT_TIME)
except Empty:
    #-- Ok...the queue is empty and it's been MAX_WAIT_TIME since
    #-- We've pulled anything from the queue.
    p.terminate( )  #-- "terminate with extreme prejudice"
else: # got line
    #-- Got a (timestamp, line_of_text) pair, where the timestamp is the
    #-- system time when the I/O polling thread grabbed the line from
    #-- the subprocess pipe.  This timestamp isn't strictly necessary,
    #-- but might come in handy in debugging the brain-dead Spynner process.
    #-- So now...do something with that line of text!
    doSomething(line)

You'll have to augment this code with some logic for spawning a new Spynner process to pick up where the terminated one left off, etc., but hopefully this should give you an idea of how to proceed.