Why is my threading/multiprocessing python script not exiting properly?

Question

I have a server script that I need to be able to shutdown cleanly. While testing the usual try..except statements I realized that Ctrl-C didn't work the usual way. Normally I'd wrap long running tasks like this

try:
    ...
except KeyboardInterrupt:
    #close the script cleanly here

so the task could be shutdown cleanly on Ctrl-C. I have never ran into any problems with this before, but somehow when I hit Ctrl-C when this particular script is running the script just exits without catching the Ctrl-C.

The initial version was implemented using Process from multiprocessing. I rewrote the script using Thread from threading, but same issue there. I have used threading many times before, but I am new to the multiprocessing library. Either way, I have never experienced this Ctrl-C behavior before.

Normally I have always implemented sentinels etc to close down Queues and Thread instances in an orderly fashion, but this script just exits without any response.

Last, I tried overriding signal.SIGINT as well like this

def handler(signal, frame):
    print 'Ctrl+C'

signal.signal(signal.SIGINT, handler)
...

Here Ctrl+C was actually caught, but the handler doesn't execute, it never prints anything.

Besides the threading / multiprocessing aspect, parts of the script contains C++ SWIG objects. I don't know if that has anything to do with it. I am running Python 2.7.2 on OS X Lion.

So, a few questions:

1. What's going on here?
2. How can I debug this?
3. What do I need to learn in order to understand the root cause?

PLEASE NOTE: The internals of the script is proprietary so I can't give code examples. I am however very willing to receive pointers so I could debug this myself. I am experienced enough to be able to figure it out if someone could point me in the right direction.

EDIT: I started commenting out imports etc to see what caused the weird behavior, and I narrowed it down to an import of a C++ SWIG library. Any ideas why importing a C++ SWIG library 'steals' Ctrl-C? I am not the author of the guilty library however and my SWIG experience is limited so don't really know where to start...

EDIT 2: I just tried the same script on a windows machine, and in Windows 7 the Ctrl-C is caught as expected. I'm not really going to bother with the OS X part, the script will be run in an Windows environment anyway.

Answer 1

This might have to do with the way Python manages threads, signals and C calls.

In short - Ctrl-C cannot interrupt C calls, since the implementation requires that a python thread will handle the signal, and not just any thread, but the main thread (often blocked, waiting for other threads).

In fact, long operations can block everything.

Consider this:

>>> nums = xrange(100000000)
>>> -1 in nums
False (after  ~ 6.6 seconds)
>>>

Now, Try hitting Ctrl-C (uninterruptible!)

>>> nums = xrange(100000000)
>>> -1 in nums
^C^C^C   (nothing happens, long pause)
...
KeyboardInterrupt
>>>

The reason Ctrl-C doesn't work with threaded programs is that the main thread is often blocked on an uninterruptible thread-join or lock (e.g, any 'wait', 'join' or just a plain empty 'main' thread, which in the background causes python to 'join' on any spawned threads).

Try to insert a simple

while True:
    time.sleep(1)

in your main thread.

If you have a long running C function, do signal handling in C-level (May the Force be with you!).

This is largely based on David Beazley's video on the subject.

Answer 2

It exits because something else is likely catching the KeyboardInterupt and then raising some other exception, or simply returning None. You should still get a traceback to help debug. You need to capture the stderr output or run your script with the -i commandline option so you can see traceback. Also, add another except block to catch all other exceptions.

If you suspect the C++ function call to be catching the CTRL+C try catching it's output. If the C function is not returning anything then there isn't much you can do except ask the author to add some exception handling, return codes, etc.

try:
    #Doing something proprietary  ...
    #catch the function call output
    result = yourCFuncCall()

    #raise an exception if it's not what you expected
    if result is None:
        raise ValueError('Unexpected Result')

except KeyboardInterupt:
    print('Must be a CTRL+C')
    return

except:
    print('Unhandled Exception')
    raise

Answer 3

how about atexit? http://docs.python.org/library/atexit.html#module-atexit