Ruby and Rails Async

Question

I need to perform long-running operation in ruby/rails asynchronously. Googling around one of the options I find is Sidekiq.

class WeeklyReportWorker
  include Sidekiq::Worker

  def perform(user, product, year = Time.now.year, week = Date.today.cweek)
    report = WeeklyReport.build(user, product, year, week)
    report.save
  end
end

# call WeeklyReportWorker.perform_async('user', 'product')

Everything works great! But there is a problem.

If I keep calling this async method every few seconds, but the actual time heavy operation performs is one minute things won't work.

Let me put it in example.

5.times { WeeklyReportWorker.perform_async('user', 'product') }

Now my heavy operation will be performed 5 times. Optimally it should have performed only once or twice depending on whether execution of first operaton started before 5th async call was made.

Do you have tips how to solve it?

Answer 1

Here's a naive approach. I'm a resque user, maybe sidekiq has something better to offer.

def perform(user, product, year = Time.now.year, week = Date.today.cweek)
  # first, make a name for lock key. For example, include all arguments
  # there, so that another perform with the same arguments won't do any work
  # while the first one is still running
  lock_key_name = make_lock_key_name(user, product, year, week)
  Sidekiq.redis do |redis| # sidekiq uses redis, let us leverage that
    begin
      res = redis.incr lock_key_name
      return if res != 1 # protection from race condition. Since incr is atomic, 
                         # the very first one will set value to 1. All subsequent
                         # incrs will return greater values.
                         # if incr returned not 1, then another copy of this 
                         # operation is already running, so we quit.

      # finally, perform your business logic here
      report = WeeklyReport.build(user, product, year, week)
      report.save
    ensure
      redis.del lock_key_name # drop lock key, so that operation may run again.
    end
  end
end

Answer 2

I am not sure I understood your scenario well, but how about looking at this gem:

https://github.com/collectiveidea/delayed_job

So instead of doing:

5.times { WeeklyReportWorker.perform_async('user', 'product') }

You can do:

5.times { WeeklyReportWorker.delay.perform('user', 'product') }

Out of the box, this will make the worker process the second job after the first job, but only if you use the default settings (because by default the worker process is only one).

The gem offers possibilities to:

• Put jobs on a queue;
• Have different queues for different jobs if that is required;
• Have more than one workers to process a queue (for example, you can start 4 workers on a 4-CPU machine for higher efficiency);
• Schedule jobs to run at exact times, or after set amount of time after queueing the job. (Or, by default, schedule for immediate background execution).

I hope it can help you as you did to me.