How can I run synchronous long running operations in EventMachine without blocking the reactor?

Question

I'd like to run a series of Procs, in a specified order (i.e., they can't run asynchronously). Some of them may take an arbitrarily long amount of time.

My code is running within a context of an EventMachine reactor. Is there a known idiom for writing this kind of code without blocking the main reactor?

Answer 1

As @maniacalrobot said, using EM.defer/deferrable lets the procs be run without blocking the reactor. But then you enter "callback hell" when you need to run several procs serially.

I know two solutions to make the code more readable: promises and fibers.

Promises gives you a nice API to compose asynchronous calls, there are a lot of good articles out there, including:

• http://domenic.me/2012/10/14/youre-missing-the-point-of-promises/
• http://www.html5rocks.com/en/tutorials/es6/promises/

Fibers are a more ruby specific tool which makes your code look synchronous while doing asynchronous things.

Here is an helper method to execute a proc asynchronously (deferred) but still block the calling code without blocking the main reactor (that's the magic of Fibers):

def deferring(action)
  f = Fiber.current

  safe_action = proc do
    begin
      res = action.call
      [nil, res]
    rescue => e
      [e, nil]
    end
  end

  EM::defer(safe_action, proc { |error, result| f.resume([error, result]) })

  error, result = Fiber.yield

  raise error if error

  result
end

Example of usage:

action1_res = deferring(proc do 
  puts 'Async action 1'
  42
end

begin
  deferring(proc do
    puts "Action1 answered #{action1_res}"
    raise 'action2 failed'
  end)
rescue => error
  puts error
end

Answer 2

Any code that would normally block the main reactor loop should be run using EM#defer. EM#defer takes two blocks as arguments, the first block is run in a different thread and should not block the reactor. A second, optional block, can be passed, which will be called when the first has completed (it will also receive the result of the first block).

Further reading https://github.com/eventmachine/eventmachine/wiki/EM::Deferrable-and-EM.defer

An example of chaining 2 long running operations would look like this:

logic_block = Proc.new { long_running_operation }
callback = Proc.new { |long_running_operation_result| EM.defer next_long_running_operation }

EM.defer logic_block, callback

Beware, The second (callback) block is run on the reactor loop, so if you're plan on chaining multiple blocks of long running code together, you'll need to call EM.defer again inside callbacks.