complex data manipulation in node.js using child processes

Question

I am new to node.js and in the position where I am using Node.js for a single page app. I have to occasionally do some very complex data manipulation (for AI and statistical functions). These services are written in Python. I am trying to figure out the best way to hand off these computations to a child process so my main thread is not blocked and I can use the abundance of python libraries available.

I need to be able to pass complex data (a JSON object is fine) to the child process and receive complex output (again a JSON object is ideal). Referring to Node.js math computation - child process and complex data?, this is simple enough with another node.js process using child_process.fork() and .send().

Is there a similar method in child_process.spawn()? Or is it possible to use .fork() to execute a python script. All I can figure out is how to execute system commands which don't allow but the most simple arguments. Even if I can send the data, I am unclear about how to receive data using .spawn().

I couldn't find this answer on SO, but I admit node.js is new to me.

Thank you.

Answer 1

The spawn method returns a ChildProcess object, which gives you access to the stdin and stdout of the process.

It would look something like this:

var child = require('child_process').spawn('python', ['script.py']);
child.stdout.on('data', function(data)
{
    response = JSON.parse(data);
    console.log(response);
});
child.stdin.write(JSON.stringify(data));

Answer 2

As reptilicus noted, use of ZeroMQ for integrating node.js and Python is an option.

In my SO answer is an example of easy client server application communicating over ZeroMQ. Implementation for client part in node.js shall be very similar, node.js for zeromq binding exists and complexity (or simplicity) of node.js side code is comparable to the Python one. (note, the link does not provide exact solution as in Python, it just illustrates how simple it is to do similar task).