How to trap errors in Whenever cron jobs?

Question

We have a number of Rake and Runner scripts that are executed by cron via the Whenever gem. We recently found one of them failed for more than a day and we didn't notice it. Is there a good way to catch and report on errors in background jobs? We have NewRelic for monitoring, but it doesn't seem to detect these problems.

Ideally I'd love a solution I can implement once that magically applies to all Whenever/cron Rake/Runner scripts.

Answer 1

For rake tasks specifically, you can install the newrelic-rake gem. This gem should allow you to record errors that happen in any rake tasks (though it will not capture errors in 'rails runner' invocations).

For rails runner invocations, you can define a simple trace wrapper, and wrap each 'runner' invocation in your schedule.rb file in that wrapper.

For example, if you have the following in config/initializers/trace_wrapper.rb:

require 'newrelic_rpm'

module TraceWrapper
  extend NewRelic::Agent::Instrumentation::ControllerInstrumentation
  def self.trace(task_name)
    perform_action_with_newrelic_trace(name: task_name, category: :task) do
      yield
    end
  end
end

...then you can change your 'runner' invocations in schedule.rb to be wrapped in this call as follows:

every 1.minutes do
  runner "TraceWrapper.trace('compile models') { MyModel.compile }"
end

You'll need to do this for every rails runner invocation in your schedule.rb file.

Answer 2

I used the answer from user1816629, but slightly modified it to make the TraceWrapper code a bit more concise. My schedule.rb looks like:

every 1.minutes do
  runner "TraceWrapper.trace 'MyModel.compile'"
end

And here is TraceWrapper:

module TraceWrapper
  extend NewRelic::Agent::Instrumentation::ControllerInstrumentation

  def self.trace(operation)
    perform_action_with_newrelic_trace(:name => operation, :category => :task) do
      eval operation
    end
  end
end