There's a fork of multiprocessing
called pathos (note: use the version on github) that doesn't need starmap
or helpers or all of that other stuff -- the map functions mirror the API for python's map, thus map can take multiple arguments. With pathos
, you can also generally do multiprocessing in the interpreter, instead of being stuck in the __main__
block. pathos
is due for a release, after some mild updating -- mostly conversion to python 3.x.
Python 2.7.5 (default, Sep 30 2013, 20:15:49)
[GCC 4.2.1 (Apple Inc. build 5566)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathos.multiprocessing import ProcessingPool
>>> pool = ProcessingPool(nodes=4)
>>>
>>> def func(g,h,i):
... return g+h+i
...
>>> p.map(func, [1,2,3],[4,5,6],[7,8,9])
[12, 15, 18]
>>>
>>> # also can pickle stuff like lambdas
>>> result = pool.map(lambda x: x**2, range(10))
>>> result
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>
>>> # also does asynchronous map
>>> result = pool.amap(pow, [1,2,3], [4,5,6])
>>> result.get()
[1, 32, 729]
>>>
>>> # or can return a map iterator
>>> result = pool.imap(pow, [1,2,3], [4,5,6])
>>> result
<processing.pool.IMapIterator object at 0x110c2ffd0>
>>> list(result)
[1, 32, 729]