Cluster with Express.js and Kue.js

Question

I am currently working on a project for myself which required to process stuff in the background, however, i need to communicate between Express and Kue. But a bit more about the current setup: My Express.js forks itself over half the CPUs inside the host machine. This works everything as intended. Now i run another node process which runs kue.js for background procession. Since kue schedules its jobs over a redis my main issue now is how I am able to send data from the processed background job back to my main node.js app.

Short outline of my setup:

app.js (running with node app.js)

var cluster = require('cluster'), Datasets = require('./models/datasets'), kue = require('kue'), jobs = cue.createQueue();
if(cluster.isMaster) {
  // Forking going on here    
} else {
  // Express.js app runs here
  app.get('/', function(req, res) {
    // Now here is where i schedule the job and i would like to retrieve an answer from the job when its done, like with on('complete', ...) but that doesn't work at all
    jobs.create('stuff', {lorem: 'ipsum', dolor: ''}).save();
  });
}

background.js (running also with node background.js, this is not the same node process like app)

// omiting libraries right now, its simply kue and datasets like in app.'s
jobs.process('stuff', function(job, done){
  //processing some background stuff here
  // Now here i would like to pass back an array of objects to the app.js process after the job is completed. 
}); 

Anyone got an idea for this? Every help is appreciated.

Sincerly, Crispin

Answer 1

Well, solved the problem on my own after some hours of work and testing. Here's the solution I came up with:

app.js - Forking Cluster like you usually would do. However, in a child process of cluster, run a child process of the background.js

// FOrking and stuff
} else {
  var background = child_process.fork(__dirname + '/background.js');
  app.get('/', function(req, res) {
    var job = {//job stuff here, like type of job and data and stuff};
    background.send(job);
  });

background.js - running as child process of a cluster worker

// Pretty much the same like before, but now queueing jobs and processing them too
process.on('message', function(object) {
  jobs.create(//accessing your job data here like type of job priority and data);
});

Everything works as intended, but may not be the perfect solution. However, this makes clear how you can easily send messages between background job processes and app processes omitting the need to store the data between those two. Hope this will help somebody in the future.

So long.