Node.js spawn EMFILE

Question

I am trying to run a command inside a async.forEach loop using ChildProcess.exec in my node job. here is the code

async.forEach( docPaths, function(docPath, callback) { 
 var run = [];
 // some command using docPath variable here..
 run.push(command);
debugger;
 exec(run.join(' '), function(error, stdout, stderr){
    callback();
  });
 }, callback);

Here is the error

"stack":"Error: spawn EMFILE\
at errnoException (child_process.js:478:11)\
at ChildProcess.spawn (child_process.js:445:11)\
at child_process.js:343:9\
at Object.execFile (child_process.js:253:15)\
at child_process.js:220:18\

a quick google shows i need to set ulimit value to increase the number of file descriptors can be open. some thing like "ulimit -n 10000".. (from link below)

https://groups.google.com/forum/#!topic/nodejs/jeec5pAqhps

where can i increase this..? or is there any other solution to circumvent the issue?

Appreciate your help.. Thanks much !!

Answer 1

First of all its not advisable to mess with ulimit, as it may have system wide impacts.

Instead since you are already using async, it comes with a limit paramater which you can use to limit the number of parallely executions.

async.eachLimit( docPaths, 100, function(docPath, callback) { 
 var run = [];
 // some command using docPath variable here..
 run.push(command);
debugger;
 exec(run.join(' '), function(error, stdout, stderr){
    callback();
  });
 }, callback);

Please do trial and error and replace 100 with suitable value.