You'll never have more processes running than there are workers in your pool (in your case mp.cpu_count() - 1
. If you call apply_async
and all the workers are busy, the task will be queued and executed as soon as a worker frees up. You can see this with a simple test program:
#!/usr/bin/python
import time
import multiprocessing as mp
def worker(chunk):
print('working')
time.sleep(10)
return
def main():
pool = mp.Pool(2) # Only two workers
for n in range(0, 8):
pool.apply_async(worker, (n,))
print("called it")
pool.close()
pool.join()
if __name__ == '__main__':
main()
The output is like this:
called it
called it
called it
called it
called it
called it
called it
called it
working
working
<delay>
working
working
<delay>
working
working
<delay>
working
working