Join a group of python processes with a timeout

Question

I'm using python's multiprocessing library to create several processes.

from multiprocessing import Process
processes = [Process(target=function) for function in FUNCTIONS]
for p in processes:
    p.start()

I want them to run for some duration and then if they have not completed, terminate them.

DURATION = 3600

A bad way to do it is as follows (bad because if the processes finish faster than DURATION, it still waits for all of DURATION):

from time import sleep
sleep(duration)
for p in processes:
    p.join(0)
    p.terminate()

Another bad way to do it (bad because it can possibly take N * DURATION to finish, where N is the number of processes):

for p in processes:
    p.join(DURATION)
    p.terminate()

What is a good way to do this?

Answer 1

I believe this does what you want without any polling required, and it will only wait up to your specified DURATION.

time_waited = 0
then = time.time()
for p in processes:
    if time_waited >= DURATION:
        p.join(0)
        p.terminate()
    p.join(DURATION - time_waited)
    time_waited = time.time() - then

Answer 2

This will poll every second for whether all of the processes are completed, up to DURATION. If either all processes have finished or DURATION has occurred, it joins/kills all of the processes. It's not perfect, in that it will take slightly longer than a second in each iteration of the for loop, but it will be pretty close.

from time import sleep
for _ in range(DURATION):
    if not any(i.is_alive() for i in processes): break
    sleep(1)
for p in processes:
    p.join(0)
    p.terminate()

Answer 3

The easiest thing to do is probably to have a "master" thread which join(0)s all the processes and then exits, and have the main thread join(3600) the master thread.

def wait_func():
    for p in processes:
    p.join()

wait_process = Process(target=wait_func)
wait_process.start()
wait_process.join(DURATION)
for p in processes:
    p.terminate()