In Python multiprocessing, how could different timeouts be specified for different functions passed asynchronously to a process pool? [closed]

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist

Closed 8 years ago.

Improve this question

I am trying to add multiprocessing to some code which features functions that I can not modify. I want to submit these functions as jobs to a multiprocessing pool asynchronously. So, given that I cannot modify the functions, how could I specify different timeouts for different functions passed asynchronously to a multiprocessing process pool? Thanks for any suggestions!

---

EDIT: This question is not requesting code; it is requesting suggestions, general guidance. A minimal understanding of the problem under consideration is demonstrated (note the correct use of the terms "multiprocessing", "pool" and "asynchronously"). Regarding attempted solutions, how can one present attempted solutions if one does not know how to proceed?

Further, as the first response explains, there appears to be no way to accomplish that which is asked. In other words, any attempt at a solution would fail.

Answer 1

There is really no way to put timeouts on tasks in multiprocessing pools, or to abort them once they've started, in the first place. You can only terminate the entire pool.

Which obviously means there's also no way to put different timeouts on each task.

The only way to really do that is to run each one in a process that you can kill if it oversteps the timeout.

The simplest way to do that is to have a thread in the main process for each child process, so the thread can block on proc.join(timeout), then call proc.terminate() if proc.is_alive() is still true.

The fact that you have to use a Process rather than a Pool or a ProcessPoolExecutor means you have to pass return values back manually, which is a pain. To avoid that, you could use a single-process pool/executor, submit the single job, wait for the AsyncResult/future with a timeout, and terminate the pool/executor if it times out, but that seems a little clumsy for different reasons.

Either way, once you've got threads that can wait on single-process tasks with a timeout, you just toss the threads into a pool/executor of ncpu workers and let it do the work for you.