multiprocessing.Pool.apply_async on Windows

Question

I'm trying to use a pool to farm out some subprocess calls in parallel. Everything works fine if I construct an entire iterable for the the pool and use imap, map, imap_unordered, etc. but I can't get apply_async to work.

For example, this works correctly:

from subprocess import check_call
from multiprocessing import Pool

def dispatch_call(file_name):
    return check_call(...)

if __name__ == '__main__':
    files = (constructed file list)
    pool = Pool()
    pool.imap(dispatch_call, files)
    pool.close()
    pool.join()

This, however, does not:

from subprocess import check_call
from multiprocessing import Pool

def dispatch_call(file_name):
    return check_call(...)

if __name__ == '__main__':
    files = (constructed file list)
    pool = Pool()
    for f in files:
        pool.apply_async(dispatch_call, f)
    pool.close()
    pool.join()

I checked the docs for multiprocessing, and none of the Windows specific issues seem to be relevant here. Am I just out of luck trying to get this to work on Windows?

Answer 1

You shoud pass the callback arguments as a sequence (list / tuple).

str object is also a sequence, but passing a string cause passing multiple arguments to the callback (each character of the string is treated as argument):

pool.apply_async(dispatch_call, [f])

OR

pool.apply_async(dispatch_call, (f,))