Too Many Open Files Parallel Python Subprocess error

Question

Questions with similar issue: Parallel Python - too many files and Python too many open files (subprocesses)

I am using Parallel Python [V1.6.2] to run tasks. The task processes an input file and outputs a log/report. Say, there are 10 folders each with 5000 ~ 20000 files which are read in parallel, processed and logs written out. Each file is approx 50KB ~ 250KB

After ~6 Hours of running, Parallel Python fails with the following error.

  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
  File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
  File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in  apport_excepthook
ImportError: No module named fileutils

Original exception was:
Traceback (most recent call last):
  File "PARALLEL_TEST.py", line 746, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 342, in __init__
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 506, in set_ncpus
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 140, in __init__
  File "/usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py", line 146, in start
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
  File "/usr/lib/python2.7/subprocess.py", line 1135, in _execute_child
  File "/usr/lib/python2.7/subprocess.py", line 1091, in pipe_cloexec
OSError: [Errno 24] Too many open files

While I understand, this could be the issue in subprocess pointed out here http://bugs.python.org/issue2320, but, seems the solution is only part of Py V3.2. I am currently tied to Py V2.7.

I would like to know if the following suggestion helps: [1]http://www.parallelpython.com/component/option,com_smf/Itemid,1/topic,313.0

*) Adding worker.t.close() in destroy() method of /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/pp.py

*) Increasing BROADCAST_INTERVAL in /usr/local/lib/python2.7/dist-packages/pp-1.6.2-py2.7.egg/ppauto.py

I would like to know if there is a fix available/Work Around for this issue in Python V2.7.

Thanks in Advance

Answer 1

I had left at some lines to destroy the job servers. job_server.destroy() fixes the issue.

Answer 2

My team recently stumbled upon a similar issue with the same file handle resource exhaustion issue while running celeryd task queue jobs. I believe the OP has nailed it and it's most likely the messy code in suprocess.py lib in Python 2.7 and Python 3.1.

As suggested in , Python Bug#2320, please pass in close_fds=True everywhere you call subprocess.Popen(). In fact they make that a default in Python 3.2 while also fixing the underlying race condition issue. See more details in that ticket.