Fetching the result of a Kue job and pushing this to the client over open connection

Question

I have an API endpoint that serves some JSON from a MongoDB. Simply like so:

router.get('/api/links', function (req, res) {
  // Find existing links
  links.find({ feed: 1 }, function (err, links) {
    res.json(links)
  })
})

I would like this endpoint to trigger a Kue job, and when this job is finished, I would like to somehow push the result of the job (new links in the database) to the client – something like the Twitter stream API, which keeps an open HTTP GET request.

router.get('/api/links', function (req, res) {
  // Find existing links
  links.find({ feed: 1 }, function (err, links) {
    res.json(links)
  })
  // Kue job to update the links database
  jobs.create('update subscriptions', {
    title: req.user.username,
    feeds: feeds
  }).save()
  // When the job is done, the new links in the database (the output
  // of the Kue job) should be pushed to the client
})

I'm not sure, however, how to get the result of the Kue job, nor how I should push this newly received data to the client when the job is done.

If there is no way to get the result of a Kue job, I could just query the database for new documents. However, I'm still not sure how to send another response to the client. Hoping someone could point me in the right direction!

Answer 1

actually this is covered in the documentation - https://github.com/LearnBoost/kue

"Job Events

Job-specific events are fired on the Job instances via Redis pubsub. The following events are currently supported:

• failed the job has failed
• complete the job has completed
• promotion the job (when delayed) is now queued

• progress the job's progress ranging from 0-100 For example this may look something like the following:

  var job = jobs.create('video conversion', {
  
      title: 'converting loki\'s to avi'
    , user: 1
    , frames: 200
  
  });
  
  job.on('complete', function(){
      console.log("Job complete");
  }).on('failed', function(){
      console.log("Job failed");
  }).on('progress', function(progress){
      process.stdout.write('\r  job #' + job.id + ' ' + progress + '% complete');
  });
  

bare in mind that your job might not be processed immediatly (depends on your queue), so the client can wait some time for a result..

EDIT: as mentioned in the comments, a job doesn't return any results so you should store the result in the database along with the job id and query the database when the job is complete.

in order to keep the connection open, use res.write and res.end instead of res.json which ends the connection (You'll have to JSON.stringify the data yourself). also, remember that the browser can timeout if this takes too long..

Answer 2

One comment for anybody who happens across this. I had a very similar concern when using 'kue' and I was quite surprised to find that although it supported messages like 'complete', 'failure' and 'progress', it couldn't relay results back to the requester.

Digging through the 'kue' source, I found that it was actually very easy to send your own custom messages (like a 'result' message). What I did looks like this:

var events = require('kue/lib/queue/events');

...

    queue.process(task, function(job, done) {
      console.log("Did job: "+job.id);
      events.emit(job.id, "result", "Yeah baby!");
      done();
    });

With this, you can then do:

job.on("result", function(res) {
    console.log("Got result: "+res);
});

But I hate poking into private APIs so I submitted a pull request to expose this a little bit more in 'kue' by adding a 'send' method to the job object.

In that case, the syntax is a bit clearer i.e.,

    queue.process(task, function(job, done) {
      console.log("Did job: "+job.id);
      job.send("result", "Yeah baby!");
      done();
    });