How to identify the cause in Python of code that is not interruptible with a CTRL +C

Question

I am using requests to pull some files. I have noticed that the program seems to hang after some large number of iterations that varies from 5K to 20K. I can tell it is hanging because the folder where the results are stored has not changed in several hours. I have been trying to interrupt the process (I am using IDLE) by hitting CTRL + C to no avail. I would like to interrupt instead of killing the process because restart is easier. I have finally had to kill the process. I restart and it runs fine again until I have the same symptoms. I would like to figure out how to diagnose the problem but since I am having to kill everything I have no idea where to start.

Is there an alternate way to view what is going on or to more robustly interrupt the process?

I have been assuming that if I can interrupt without killing I can look at globals and or do some other mucking around to figure out where my code is hanging.

Answer 1

In case it's not too late: I've just faced the same problems and have some tips

First thing: In python most waiting apis are not interruptible (ie Thread.join(), Lock.acquire()...). Have a look at theese pages for more informations: http://snakesthatbite.blogspot.fr/2010/09/cpython-threading-interrupting.html http://docs.python.org/2/library/thread.html

Then if a thread is waiting on such a call, it cannot be stopped. There is another thing to know: if a normal thread is running (or hanged) the main program will stay indefinitely untill all threads are stopped or the process is killed.

To avoid that, you can make the thread a daemon thread: Thread.daemon=True before calling Thread.start().

Second thing, to find where your program is hanged, you can launch it with a debugger but I prefer logging because logs are always there in case its to late to debug.

Try logging before and after each waiting call to see how much time your threads have been hanged. To have high quality logs, uses python logging configured with file handler, html handler or even better with a syslog handler.