Multiple delayed_jobs roles with Capistrano?

Question

I have a question that I am not finding much useful information for. I'm wondering if this is possible and, if so, how to best implement it.

We are building an app in Rails which has heavy data-processing in the background via DelayedJob (…it is working well for us.)

The app runs in AWS and we have a few different environments configured in Capistrano.

When we have heavy processing loads, our DelayedJob queues can back up--which is mostly fine. I do have one or two queues that I'd like to have a separate node tend to. Since it would be ignoring the 'clogged' queues, it would keep tending its one or two queues and they would stay current. For example, some individual jobs can take over an hour and I wouldn't want a forgotten-password-email delivery to be held up for 90 minutes until the next worker completes a task and checks for a priority job.

What I want is to have a separate EC2 instance that has one worker launched that tends to two different, explicit queues.

I can do this manually on my dev machine by launching one or two workers with the '--QUEUES' option.

Here is my question, how can I define a new role in capistrano and tell that role's nodes to start a different number of workers and tend to specific queues? Again, my normal delayed_jobs role is set to 3 workers and runs all queues.

Is this possible? Is there a better way?

Presently on Rails 3.2.13 with PostgreSQL 9.2 and the delayed_job gem.

Answer 1

Try this code - place it in deploy.rb after requiring default delayed_job recipes.

# This overrides default delayed_job tasks to support args per role
# If you want to use command line options, for example to start multiple workers,
# define a Capistrano variable delayed_job_args_per_role:
#
#   set :delayed_job_args_per_role, {:worker_heavy => "-n 4",:worker_light => "-n 1" }
#
# Target server roles are taken from delayed_job_args_per_role keys.
namespace :delayed_job do

  def args_per_host(host)
    roles.each do |role|
      find_servers(:roles => role).each do |server|
        return args[role] if server.host == host
      end
    end
  end

  def args
    fetch(:delayed_job_args_per_role, {:app => ""})
  end

  def roles
    args.keys
  end

  desc "Start the delayed_job process"
  task :start, :roles => lambda { roles } do
    find_servers_for_task(current_task).each do |server|
      run "cd #{current_path};#{rails_env} script/delayed_job start #{args_per_host server.host}", :hosts => server.host
    end
  end

  desc "Restart the delayed_job process"
  task :restart, :roles => lambda { roles } do
    find_servers_for_task(current_task).each do |server|
      run "cd #{current_path};#{rails_env} script/delayed_job restart #{args_per_host server.host}", :hosts => server.host
    end
  end

end

P.S. I've tested it only with single role in hash, but multiple roles should work fine too.

Answer 2

In Capistrano3, using the official capistrano3-delayed-job gem, you can do this without modifying the Capistrano methods:

# If you have several servers handling Delayed Jobs and you want to configure
# different pools per server, you can define delayed_job_pools_per_server:
#
# set :delayed_job_pools_per_server, {
#   'server11-prod' => {
#     'default,emails' => 3,
#     'loud_notifications' => 1,
#     'silent_notifications' => 1,
#   },
#   'server12-prod' => {
#     'default' => 2
#   }
# }

# Server names (server11-prod, server12-prod) in :delayed_job_pools_per_server
# must match the hostnames on Delayed Job servers. You can verify it by running
# `hostname` on your servers.

# If you use :delayed_job_pools_per_server, :delayed_job_pools will be ignored.