Python multiprocessing Pool on Windows 8.1 spawns only one worker

Question

I currently have this piece of code (feel free to comment on it too :) )

def threaded_convert_to_png(self):
    paths = self.get_pages()
    pool = Pool()
    result = pool.map(convert_to_png, paths)
    self.image_path = result

On an Intel i7 it spawns eight workers when running on Linux; however, when running Windows 8.1 Pro it only spawns one worker. I checked and cpu_count() returns 8 on both Linux and Windows.

• Is there something I am missing here, or doing wrong?
• Is there a way to fix that problem?

P.S. This is in Python 2.7.6

Answer 1

There is one easy way to determine what is happends in your pool - to turn on multiprocessing debug. You can do it like this:

import logging
from multiprocessing import util

util.log_to_stderr(level=logging.DEBUG)

And on script running you will get full info about processes running, spawning and exiting.

But any way, process pool always spawn N processes (where is N - "processes" argument value or cpu_count), but tasks distribution between processes can be uneven - it depends on task run time.

Answer 2

I managed to solve my similar problem. I'm not sure if it's help for you but I decided to document it here anyway in case it helps someone.

In my case I was analyzing huge amount of tweets (52000 in total) by dividing them to multiple processors. It worked fine on OSX and on server, but on my Windows 8.1 it was really slow and processes were activated sequentially. By looking into task-manager I noticed that the main Python process' memory usage went up and up to around 1.5Gb. The worker process' memory usage climbed similarly. Now I noticed that my older version worked fine which had slightly different algorithm. In the end the problem was that I retrieved whole tweets from database while I required only the text part of the tweets. This apparently led to grown memory usage. After I fixed that part, the program launched worker processes properly.

So based on my experience I have a hunch that Windows tries to control the ram usage by blocking the worker processes. If so, check the ram usage of your processes. This is just speculation on my part, so I'm interested if someone has better explanation.