There is really no way to put timeouts on tasks in multiprocessing pools, or to abort them once they've started, in the first place. You can only terminate the entire pool.
Which obviously means there's also no way to put different timeouts on each task.
The only way to really do that is to run each one in a process that you can kill if it oversteps the timeout.
The simplest way to do that is to have a thread in the main process for each child process, so the thread can block on proc.join(timeout)
, then call proc.terminate()
if proc.is_alive()
is still true.
The fact that you have to use a Process
rather than a Pool
or a ProcessPoolExecutor
means you have to pass return values back manually, which is a pain. To avoid that, you could use a single-process pool/executor, submit the single job, wait for the AsyncResult
/future
with a timeout, and terminate
the pool/executor if it times out, but that seems a little clumsy for different reasons.
Either way, once you've got threads that can wait on single-process tasks with a timeout, you just toss the threads into a pool/executor of ncpu workers and let it do the work for you.