How to find out if a set of Resque jobs have finished?

Question

I have the following code:

[23, 45, 69, 20].each do |page_id|
  Resque.enqueue(ProcessPage, page_id)
end

I would like to know when all those jobs have finished processing to notify the user via email. My first attempt was to notify the user at the very end of the job, but that would work just individually for each job.

I have found this gem resque-status, which basically let's you do this:

job_ids = []
[23, 45, 69, 20].each do |page_id|
  job_ids << ProcessPage.create(page_id)
end

And then you can check the status of a job by doing:

status = Resque::Plugins::Status::Hash.get(job_id)

What confuses me is... When should I do that checking for all the statuses? I mean, would I be doing a while checking the status of all the jobs? That would make the server to timeout, any ideas how I can do this?

Answer 1

I'm not going to paste any code here because it looks like you're pretty familiar with how jobs should be written but think about the following idea:

You should have a resque job which gets and array of ids to process. This resque jobs creates a single resque job for each of the ids so they will be processed in parallel. After creating the resque jobs for specific ids it will create another resque jobs which gets the list of ids and periodically checks on them to see if they're finished. When they're done, this job will send the user an email.

With this paradigm you enjoy all of the worlds:

1. Each ID has its own job.
2. Parallel execution.
3. Status check is done on the background thus the user won't get timeout.

UPDATE:

A pseudo code for the check resque job:

class CheckerJob
  @queue = :long

  def self.perform(ids)
    finished = []
    while finished.size < ids.size
      ids.each do |id|
        finished << id if job_finished?(id)
      end
      sleep 10
    end
    send_email_to_user
  end
end

Now all you left to do is to implement both job_finished?(id) and send_email_to_user methods.

Answer 2

Easy solution would be to have your job take a list of ids:

Class Job
  def self.perform(ids)
    threads = ids.inject([]) do |memo, id|
      memo << Thread.new { # whatever }
    end
    ThreadsWait.join_all(*threads)
    # All threads are done
    # Notify user that job is finished
  end
end

Don't have a job that checks whether this one has finished. If you're going to go that kind of route, subscribe to Redis events http://redis.io/topics/internals-rediseventlib

UPDATE: added multithreading

Answer 3

How about doing it this way?

Resque.enqueue_to("page_processes#{user.id}", ProcessPage, page_id)

#code in ProcessPageWorker below
def self.perform(page_id, user_id)
  #work
  # code below notifies user if the 'page_processes' queue is equal to 1
  notify_user if Resque.size("page_processes#{user_id}") == 1
end

We enqueue the job in a queue named 'page_processes' plus the user's ID. Once every job is done, we check the size of the "page_processes#{user_id}" queue. If the queue is equal to 1, it would mean that the job being worked on is the last job in the queue so we should notify the user. We include the user_id in the name of the queue to ensure the queue's unique name.

Let me know if this helps.