Python Multiprocessing; Infinite Processes

Question

I have a function in my main python file that does some multiprocessing, which works fine;

    if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=len(directories))
    pool.map(worker, directories)        

However, I imported a .py file from an other directory, in which I try to do exactly the same;

# Main file    
import multiprocessing       
read_DataFiles.test(os.getcwd())

# Imported file
directories=["x", "x", "x"]
def worker(sample):
    File=open('test'+sample+'.bat', 'w')
    File.close()
    1 == 1

def test(path):
    if __name__ == 'read_DataFiles':
        pool = multiprocessing.Pool(processes=8)
        print pool.map(worker, directories)  

which doesn't stop working, it continues to create new processes. Anyone sees what I'm doing wrong?

Answer 1

The difference is around the if __name__ == .... On Windows, multiprocessing is a hack that works by creating new processes and re-importing the code in each of them. I'm sure that you call test(path) from the top-level of another module. The check in this function, if __name__ == 'read_DataFiles':, is pointless: this is always true, which means it will always start a new pool. What you want instead is to use if __name__ == '__main__' from the main script, and only call test(path) if that is the case.