Python multiprocessing Pool recovery after "Resource temporarily unavailable"

Question

If I create a Pool with an unacceptably-high number of processes while in the Python interpreter, it will obviously error-out, however it doesn't seem like the forked processes are cleaned-up before doing so, therefore leaving the environment dirty, and the rest of the system unable to fork processes.

>>> from multiprocessing import Pool
>>> p = Pool(1000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
    self._repopulate_pool()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable

>>> p = Pool(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
    self._repopulate_pool()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool
    w.start()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 130, in start
    self._popen = Popen(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
    self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable

Is there some way to avoid/remedy this, or is it considered a bug?

Answer 1

Is there some way to avoid/remedy this,

Don't do that.

or is it considered a bug?

Yes in the sense that all resources allocated should be de-allocated if the initializer fails. You should check on the specific build of 2.7 that you are using and see if there are any multiprocessing-specific library bugs fixed in later builds (2.7.6 release notes: http://hg.python.org/cpython/raw-file/99d03261c1ba/Misc/NEWS).

I'm assuming that your platform is OSX based on the paths in the stacktrace. Here is a post on errno 35 (which appears to be EAGAIN in OSX) when forking - I can't run more than 100 processes

Whatever it is that you're trying to accomplish, it seems that you need to incorporate a limit on resource usage at the application level. That means you might need to rethink your solution. With your present solution and with the bug fixed, you'll still likely see the resource limit hit system-wide in other contexts.

Answer 2

I faced the same issue and it was able to fix it by as per Dustin's comment.

Ticket : http://bugs.python.org/issue19675

I'm using Python 2.7.8 on Mac OS Mavericks

Answer 3

In my case, you should set the "ulimit -n 2048" in the terminal which you are going to run the function. the number 2048 could be higher. It solved my problem.