Watching for multiprocessing to finish

Question

The poolDict dictionary here is used to write some updates on the processes started with multiprocessing Pool(). The while loop is running at the background and it is watching for any updates happening with poolDict. As soon as the condition is met len(poolDict)==3 while loop is stopped. The code below runs fine. But a real world program that utilizes the same approach outlined here introduces some strange behavior. The while loop for example doesn't want to stop running even after the condition is met. It takes at least 15 seconds for while loop to finally 'realize' that len(poolDict) is already equal to 3. It causes a noticeable time lag that needs to be fixed. Instead of while loop what else could be used as a replacement? What approach to take in order to make an update instant?

import time
import multiprocessing as mp
poolDict=mp.Manager().dict()

def myFunction(arg):
    print '\n\tprocessing arg', arg, '...'
    for i in range(arg+1):
        if i==arg:
            poolDict[arg]=True
            print '\n\t\t...processing completed for', arg

pool=mp.Pool(processes=2)
pool.map_async( myFunction, [15000001,15000002,15000003] )

status=True
while status:
    print status
    time.sleep(.2)
    if len(poolDict)==3:
        status=False
        print '...while loop was stopped'

print 'completed'

Answer 1

Why not use the return value of map_async?

result=pool.map_async(myFunction, ...)
result.wait()

Since the dictionary itself destroys the order of the values, I would in fact consider using imap_unordered to transport all the values, something like:

def myFunction(arg):
     # time consuming stuff...
     return arg, True

nonpooldict = dict(pool.imap_unordered(myFunction, data))
# Will collect results in a dictionary as they are calculated

Otherwise you may want to try a synchronization method such as SyncManager.Event. I'm not sure if there is a way to flush specific types of SyncManager changes like the dictionary.