Separate logs for delayed_job named queues

Question

We're using delayed_job's named queues in our project. Is there any way to log each queue into separate log file?

DelayedJob in started in deploy script like this:

run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 1 --queue=fast start"
run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 2 start"

What I'd like to achieve is that everything running in fast queue would log into log/delayed_job.fast.production.log, and everything in the other queue would log into log/delayaed_job.production.log.

Answer 1

Unfortunately, the answer by jvperrin didn't help. What we've managed to do is to set the needed log file in initializer:

Delayed::Worker.logger = ActiveSupport::BufferedLogger.new("log/delayed_job.#{Rails.env}.log", Rails.logger.level)

if caller.last =~ /script\/delayed_job/
  queues = ARGV.select { |opt| opt =~ /--queue=/ }
  queue = /\=(.*)/.match(queues.first) if queues
  queue = $1 if queue

  if queue
    Delayed::Worker.logger = ActiveSupport::BufferedLogger.new("log/delayed_job.#{queue}.#{Rails.env}.log", Rails.logger.level)
  end

  ActiveRecord::Base.logger = Delayed::Worker.logger
  Rails.logger = Delayed::Worker.logger
end

Answer 2

Try adding replacing the two commands in your deploy script with the following two commands:

run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 1 --queue=fast start >> ./log/delayed_job.fast.production.log 2>&1"
run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 2 start >> ./log/delayed_job.production.log 2>&1"

I added >> ./log/delayed_job.fast.production.log 2>&1 to the end of the first command and >> ./log/delayed_job.production.log 2>&1 to the end of the second command. These added sections will take the output of the delayed_job commands and redirect both stdout and stderr to each of the log files. The output may not be written immediately to the file, probably because there is some kind of buffer in place for file writing.

If you want the output to also continue to appear on the screen as well as being logged to file, then you can use the tee command:

run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 1 --queue=fast start | tee -a ./log/delayed_job.fast.production.log 2>&1"
run "cd #{current_path};RAILS_ENV=#{rails_env} script/delayed_job -p #{rails_env} -i 2 start | tee -a ./log/delayed_job.production.log 2>&1"

Answer 3

In latest Rails version use (based on Paul Nosoff's answer):

if caller.last =~ /bin\/delayed_job/

  queues = ARGV.select { |opt| opt =~ /--queue=/ }
  queue = /\=(.*)/.match(queues.first) if queues
  queue = $1 if queue

  if queue
    log_file_name = Rails.root.join('log', "delayed_job.#{queue}.#{Rails.env}.log")
  else
    log_file_name = Rails.root.join('log', 'delayed_job.log')
  end
  # Daily log rotate, Debug level:
  logger = ActiveSupport::Logger.new(log_file_name, 'daily', level: Logger::DEBUG)
  Rails.logger = ActiveRecord::Base.logger = Delayed::Worker.logger = logger
end