Stalker/beanstalk process jobs asynchronously

Question

Ok I'm doing some testing with Stalker and Beanstalkd. My objective is to offload 500kb post requests to queue and process them asynchronously.

So far in my testing I have this very simple example.

#worker.rb
require 'stalker'
include Stalker

job 'hello' do |args|
  puts "hi"
  sleep 1
  puts "hello"
end

and this file for adding to queue

# stalker.rb
require 'rubygems'
require 'stalker'

10.times do 
    Stalker.enqueue('hello')
    puts 'queued'
end

So in one terminal I run

$ stalk worker.rb
Working 1 jobs: [ hello ]

Then I run the stalker file

$ ruby stalker.rb

stalker executes almost instantly as expected so no blocking on that.

but the worker takes approximately 10 seconds to run. Really I want this to be much closer to 1 second for those 10 jobs because I want them to run in parallel.

Any recommendations on the next best step to do this?

UPDATE: I've realised I can run multiple workers from different terminals and it will process the queue much faster e.g. 2 workers will do the process in roughly half the time.

Answer 1

As a heads up, you may want to check out the new gem I created called Backburner which is a more modern way to manage asynchronous jobs with beanstalkd in ruby. It follows a cleaner, resque-esque interface, has a truly asynchronous worker (that uses forking and multiple threads for much faster throughput) and has much better error handling and retry support. I was using stalker for 2 years before writing backburner because I wanted a better solution.

It gives you support for things like:

SomeObject.async(:pri => 1000, :delay => 2).some_method(1, 2, 3)

with the method then being automatically enqueued onto beanstalkd and processed asynchronously by backburner workers.

Answer 2

Stalker.enqueue takes 3 options: job name string, arguments hash, options hash. in the option hash, you can pass the number of seconds that it takes to start . example:

Stalker.enqueue('hello',{},{:delay => 5})