Node.js workers/background processes

Question

How can I create and use background jobs in node.js?

I've come across two libs (node-resque and node-worker) but would like to know if there's something more used.

Answer 1

I did some research on this and I would do it like this.

Setup

beanstalkd

1. Install beanstalkd. Another message queue, BUT this one supports DELAYED PUTS. If you compile from source it is going to be a little harder because it depends on libevent(like memcached). But then again, I don't think you have to compile it from source, because there are a lot of binary packages available. For example on Ubuntu you can install beanstalkd by issuing the command:

   sudo apt-get install beanstalkd

node-beanstalk-client

2. Install a beanstalkd client library. The best one I found was node-beanstalk-client. Because on the beanstalkd client library list this library isn't/wasn't mentioned(Then again I can add entries to the list, so I will add this one). The reasons I prefer this library over the others are:

   1. Npm package: I liked to use a npm package to install client library. The others did not have any.
   2. Active development: I prefer libraries which have later/more commits.

So to install it, after you have installed npm(the write way) you would just issue the following command:

npm install beanstalk_client

Code

consumer.js

var client = require('beanstalk_client').Client;

client.connect('127.0.0.1:11300', function(err, conn) {
    var reserve = function() {
        conn.reserve(function(err, job_id, job_json) {
            console.log('got job: ' + job_id);
            console.log('got job data: ' + job_json);
            console.log('module name is ' + JSON.parse(job_json).data.name);
            conn.destroy(job_id, function(err) {
                console.log('destroyed job');
                reserve();
            });
        });
    }

    reserve();
});

First start the consumer:

node consumer.js 

Next start producer.js. Five seconds(delay I specified) after you execute producer.js, consumer.js will process the message.

producer.js

var client = require('beanstalk_client').Client;
client.connect('127.0.0.1:11300', function(err, conn) {
    var job_data = {"data": {"name": "node-beanstalk-client"}};
    var priority = 0;
    var delay = 5;
    var timeToRun = 1;
    conn.put(priority, delay, timeToRun, JSON.stringify(job_data), function(err, job_id) {
        console.log('put job: ' + job_id);
        process.exit();
    });
});

to start just issue:

node producer.js

Answer 2

I'm using Kue.

Kue is a feature rich priority job queue for node.js backed by redis. A key feature of Kue is its clean user-interface for viewing and managing queued, active, failed, and completed jobs.