How to make an HTTP request without waiting for the response in Ruby

Question 1

It's possible by opening a socket and closing it. This will establish a connection and close the connection without transferring any data within the connection's context...

...you will need to wait for the connection to open - although there might be a way to avoid that.

require 'socket'
# opens a connection, will wait for connection but not for data.
s = TCPSocket.new 'host.url.com', 80
# closes the connection
s.close

It's probably the equivalent of a ping and will not open a new thread... although, it's not totally asynchronous.

With an HTTP request, the code might look something like this:

require 'socket'
host = 'www.google.com'
# opens a connection, will wait for connection but not for data.
s = TCPSocket.new host, 80
# send a GET request for '/' .
s.puts "GET / HTTP/1.1\r\nHost: #{host}\r\n\r\n"
# closes the connection
s.close

You can search for more info about HTTP requests on stack exchange and get some ideas, like here.

Just to clarify (due to comments):

This WILL introduce the latency related to establishing the connection (and sending the request), but you WILL NOT have to wait for the reply to be processed and received.

Dropping the connection (closing your half of the socket) could have any of the following effects - all of which are assuming a decent web server:

if s.close is completed BEFORE the response is fully sent by the web server, the web server will first process the request and then an exception will be raised on the web-server's socket when it tries to send the data. The web server should then close the socket and release any resources.
if s.close is completed AFTER the response is fully sent by the web server, then the server might either: 1. close the socket immediately (normal HTTP 1 behavior) OR 2. keep the connection alive until a timeout occurs (optional HTTP 1.1 behavior) - Timeout is usually about 10 seconds.

Hitting the web server repeatedly in very small intervals might cause a DOS security flag to be raised and future connections to be blocked (this is true regardless of how you hit the web server).

I would probably opt to use a worker thread, like so:

I do believe that running a separate thread might not be as expensive as you believe. It's possible to have one thread cycle for all the asynchronous web requests.

here's a thought:

require 'socket'

REQUESTS_MUTEX = Mutex.new
REQUESTS_QUE = []
REQUESTS_THREAD = Thread.new do
   begin
      loop do
         sleep 0.5 while REQUESTS_QUE.empty?
         host, path = REQUESTS_MUTEX.synchronize {REQUESTS_QUE.shift}
         # the following will open a connection and start a request,
         # but it's easier to use the built in HTTP API...
         # although it will wait for a response. 
         s = TCPSocket.new host, 80
         s.puts "GET #{path} HTTP/1.1\r\nHost: #{host}\r\n\r\n"
         s.close
         # log here: 
         puts "requested #{path} from #{host}."
      end
   rescue Exception => e
      retry
   end
end
def asynch_request host, path = '/'
   REQUESTS_MUTEX.synchronize {REQUESTS_QUE << [host, path]}
   REQUESTS_THREAD.alive?
end

Now, for every request, you can simply call asynch_request and the cycling thread should hit the web server as soon as it wakes up and notices the que.

You can test it from the terminal by pasting a few requests in a bunch:

asynch_request 'www.google.com'
asynch_request 'www.yahoo.com'
asynch_request 'I.Dont.exist.com'
asynch_request 'very bad host address...'
asynch_request 'www.github.com'

Notice the silent fails (you can tweak the code).

Question 2

From your controller, add the request-url job to a queue.

Then run a background processes that reads from the queue and performs the requests.

This will remove the request performance delay from your controller action.

Rails 4.2 includes a way of doing this that is abstracted from the particular back-end implementation. It's called ActiveJob:

https://github.com/rails/rails/tree/master/activejob

Here's an example of using it with the Sidekiq service:

https://github.com/mperham/sidekiq/wiki/Active-Job

If you're using an older version of Rails, you could use one of the queue services directly as well.

Question 3

It is possible, but you need to use ruby eventmachine

Then you can use em-http-request to perform the async http request, i.e.:

First install the gems

gem install 'eventmachine'
gem install 'em-http-request'

Then try the code

require 'rubygems'
require 'eventmachine'
require 'em-http'

urls = %w(http://www.google.com http://www.rorra.com.ar)

pending = urls.size

EM.run do
  urls.each do |url|
    http = EM::HttpRequest.new(url).get
    http.callback {
      puts "#{url}\n#{http.response_header.status} - #{http.response.length} bytes\n"
      puts http.response

      pending -= 1
      EM.stop if pending < 1
    }
    http.errback {
      puts "#{url}\n" + http.error

      pending -= 1
      EM.stop if pending < 1
    }
  end
end

Question 4

There are several different HTTP libraries for ruby. Some of them allow "asynchronous" requests for ruby. Although usually it will be in another thread. I think you may be incorrect to say you can't do that for performance reasons.

HTTPClient is my preferred HTTP client library, although it is not neccesarily the most popular. With HTTPClient you can do:

conn = HTTPClient.new.get_async("http://example.com")

Normally you'd then check to see when the request is done using the connection object that's returned, but you can also just ignore the connection object returned. In either case, the actual HTTP request is being made in a separate thread, so your main thread is not waiting on it and can continue executing other code.

Other ruby http client libraries also offer asynchronous modes. You can also do it yourself simply by launching a thread which makes the http request, and not even waiting on the thread completion if you don't care about it. You could use some of the tools in concurrent-ruby to use thread pools and code someone else has already written to minimize worries about performance either. Perhaps a Future from concurrent-ruby.

If you really truly don't want to use threads, then you basically have to use EventMachine as someone else suggested. I would not assume this will neccesarily lead to better performance though.

There are some compiled C gems that let you make async requests where it's not obvious you are creating threads -- but the C code will still likely be creating threads in C land. Threads are basically how you can do asynchronous things like you want to do. Or complicated event/fiber based solutions like EventMachine, sure.

In Rails 4.2, you can use ActiveJob to queue up the HTTP request as a background process. This is another way to kick off an HTTP request but not have your controller wait on it. But you have to set up a back-end for ActiveJob (there are several choices), and the back-end will either be running in an entirely different process (possibly more than one).... or again, the backend will be creating threads for you.

I'd suggest considering to abandon your resistance to threads, threads are really a fine way to handle this and should not be a performance concern -- I'd probably use concurrent-ruby, and some of the higher level abstractions it gives you like Futures (which are still implemented in terms of threads under the hood), to avoid having to write the thread code directly myself, and use a library written by other people who know what they're doing and set everything up reasonably for any performance concerns. Or if you really truly want to avoid threads, I'd use ActiveJob, with a back-end adapter that does not use threads. Personally, I would not take the EventMachine route, it's adding a lot of things to deal with, just for the purpose of async http requests.

Or, yeah, just make a HEAD request and figure it's quick enough not to worry about it.

Or, sure, Myst's answer about opening a socket directly so you can immediately close it without waiting on the response seems interesting.

Question 5

If making a new process is ok (not a whole rails process mind you) you could use something along these lines: Executing shell command in background from ruby with proper argument escaping

# Spawn a new process and run the curl command
pid = Process.spawn("curl", "http://example.com", :out =>    '/dev/null', :err => '/dev/null')

# Detach the spawned process
Process.detach pid

When I benchmark this I get 1.999ms. Compared against using Process.wait pid which took 248ms

Question 6

The first thing what poped up in my mind was: perhaps detach the request somehow. Not sure if this possible with request in ruby... I know you can detach processes.

Other, not 100%, solution would be to just ask for the headers, so you transfer just a small amount of data. This thread seems to have some good hints: https://stackoverflow.com/a/9699874/1933185