python using multiprocessing pool.map for distributed computing

Question

The next code works to execute "someprogram" in parallel for the 50 inputfilenames, using a pool of 5 workers. How is it possible to use 5 workers on this computer, and 7 on another computer, using only the standard libraries like multiprocessing. Ideally i would have a list tuples (hostname, amountofworkers), which can be used to speed up something, and maybe turn it into a decorator so that it can be reused more easily on methods like the "commandlinestuff". (Using linux and python 2.7)

import multiprocessing
import subprocess

def commandlinestuff(inputfilename):
    p = subprocess.Popen("someprogram "+inputfilename, shell=True)
    p.wait()

inputfilenames = ["something"+str(i).zfill(2)+".in" for i in range(50)]
p = multiprocessing.pool(5)
p.map(commandlinestuff, inputfilenames)

Answer 1

https://code.google.com/p/distributed-python-for-scripting/ did the trick for me - nothing to install, and seems to be the shortest/easiest way possible to do multiprocessing in python

Answer 2

It sounds like you are trying to re-invent pyro which is itself written in pure python but is not currently a part of the standard libraries.

Basically you need a server running on the remote machine(s) that accepts a connection, receives a pickled object to execute, (and possibly some data), executes it and posts back a result. You will also need a client on the local machine that does the posts, gathers the results and possibly does some load balancing.

The Parallel Processing entry in the python wiki gives a long list of tools to do this sort of thing with various advantages and disadvantages.