Is it possible to initiate multiple parallel http requests using EventMachine with Ruby 1.8

Question

em-synchrony.rb implements this feature with Fibers but I would go for a non-Fibre version with 1.8 MRI.

EM.run do
  http = EM::Protocols::HttpClient2.connect("www.google.com", 80)
  request = http.get("/")
  request.callback do
    puts request.status
    EM.stop
  end
end

Answer 1

Have a look at em-http-request:

EM.run do
  http1 = EventMachine::HttpRequest.new('http://example.com/1').get
  http1.callback do
    p http1.response
  end
  http2 = EventMachine::HttpRequest.new('http://example.com/2').get
  http2.callback do
    p http2.response
  end
end

Answer 2

If you can look outside of EventMachine, Typhoeus is an easy to use HTTP client that comes with Hydra, which gives the ability to process multiple requests in parallel.

I used it for a couple things and it's easy to set up. This is some untested code torn out of something I wrote the other day:

require 'typhoeus'

hydra = Typhoeus::Hydra.new(:max_concurrency => 10)
urls.each do |url|
  request = Typhoeus::Request.new(url)

  request.on_complete do |resp|
    filename = resp.request.url.split('/').last
    puts "writing #{filename}"
    File.open(filename, 'w') do |fo|
      fo.write resp.body
    end  
  end
  puts "queuing #{ url }"
  hydra.queue(request)
end

hydra.run