long running CPU intensive python script sent to sleep by scheduler

Question

I have written a data munging script that is very CPU intensive. It has been running for a few days now, but now (thanks to trace messages sent to the console), I can see that it is not working (actually, has not been working for the last 10 hours or so.

When I run top, I notice that the process is either sleeping (S) or in uninterreptable sleep (D). This is wasting a lot of time.

I used sudo renice -10 PID to change the process's nice value, and after running for a short while, I notice that the process has gone back to sleep again.

My question(s):

1. Is there anything I can do to FORCE the script to run until it finishes (if even it means the machine is unusable until the end of the script?

2. Is there a yield command I can use in Python, which allows me to periodically pass control to other process/threads to stop the scheduler from trying to put my script to sleep?.

I am using python 2.7.x on Ubuntu 10.0.4

Answer 1

The scheduler will only put your process on hold if there is another process ready to run. If you have no other processes which hog up the CPU, your process will be running most of the time. The scheduler does not put your process to sleep just because it feels like it.

My guess is that there is some reason your process is not runnable, e.g. it is blocking and waiting for I/O or data.