node.js / ruby integration with beanstalkd

Question

This is related to another question specific to payment processing, and that is my example use case, but I was considering trying to integrate node.js and ruby on the same server using beanstalkd. Basically, I want to use node.js as my main web server, but when I need to do some payment processing, I'd like to use something robust and stable like ruby.

I was considering trying to use beanstalkd as a way to have node.js queue up payment processing jobs for ruby to perform in the background. The documentation for beanstalkd is a little slim, so I'm having trouble figuring out if this is a good approach, or exactly how I would go about it. From what I can tell though, it should be fairly straightforward to start a beanstalkd process and then have node.js connect to it to send it jobs, and have a ruby script which can perform the jobs and send back the results.

Answer 1

So after rooting around enough, I did find the documentation I really needed to evaluate beanstalkd. There is a protocol document in the source not linked to from anything I've been reading or the main page (although it is in a folder called doc) that gives better details about its capabilities and limitations.

It seems very nice as an asynchronous worker queue, which is perfect for node.js, and it would be a good fit to communicate with some Ruby code to do payment processing, but as dkam says, how do I get the response back to node.js to be able to update the client. While I think this makes sense for many tasks, it is not sufficient for mine.

Given alfred's advice, I've investigated redis, and while it isn't exactly what I need right out of the box, I think it will be sufficient. There is already an actor library out there built on top of redis for Ruby, so I think I should be able to make something simple that can talk between node and Ruby with roughly actor style semantics, or at least callback semantics.

Answer 2

Beanstalk is appropriate for this task. Make sure you use the binlog option to make the jobs persistent between beanstalkd restarts.

Your node.js processes will use a tube (called, say 'payments') and put jobs into it, with an appropriate priority.

Your Ruby script can then watch the payments tube and process the jobs.

Make sure you give the jobs an adequate TTL - you want to ensure the payment processing has time to complete before beanstalk assumes the job has failed and re-queues it.

Just curious - how will you provide feedback to the customer that the payment has succeeded? Perhaps the Ruby script will update a record in the database?