WEBrick socket returns eof? == true

Question

I'm writing a MITM proxy with webrick and ssl support (for mocking out requests with VCR on client side, see this thread VCRProxy: Record PhantomJS ajax calls with VCR inside Capybara or my github repository https://github.com/23tux/vcr_proxy), and I made it really far (in my opinion). My configuration is that phantomjs is configured to use a proxy and ignore ssl errors. That proxy (written in webrick) records normal HTTP requests with VCR. If a SSL request is made, the proxy starts another webrick server, mounts it at / and re-writes the unparsed_uri for the request, so that not the original server is called but my just started webrick server. The new started server handles then the requests, records it with VCR and so on.

Everything works fine when using cURL to test the MITM proxy. For example a request made by curl like

curl --proxy localhost:11111 --ssl --insecure https://blekko.com/ws/?q=rails+/json -v

gets handled, recorded...

But: When I try to do the same request inside page served by poltergeist from javascript with an jsonp ajax request, something goes wrong. I debugged it to the line which causes the problem. It's inside the httpserver.rb from webrick inside the ruby source code at line 80 (Ruby 1.9.3):

def run(sock)
  while true
    res = HTTPResponse.new(@config)
    req = HTTPRequest.new(@config)
    server = self
    begin
      timeout = @config[:RequestTimeout]
      while timeout > 0
        break if IO.select([sock], nil, nil, 0.5)
        timeout = 0 if @status != :Running
        timeout -= 0.5
      end
      raise HTTPStatus::EOFError if timeout <= 0
      raise HTTPStatus::EOFError if sock.eof?

The last line raise HTTPStatus::EOFError if sock.eof? raises an error when doing requests with phantomjs, because sock.eof? == true:

1.9.3p392 :002 > sock
 => #<OpenSSL::SSL::SSLSocket:0x007fa36885e090> 
1.9.3p392 :003 > sock.eof?
 => true 

I tried it with the curl command and there it's sock.eof? == false, so the error doesn't get raised, and everything works fine:

1.9.3p392 :001 > sock
 => #<OpenSSL::SSL::SSLSocket:0x007fa36b7156b8> 
1.9.3p392 :002 > sock.eof?
 => false 

I only have very little experience with socket programming in ruby, so I'm a little bit stucked.

How can I find out, what's the difference between the two requests, based on the sock variable? As I can see in the IO docs of ruby, eof? blocks until the other side sends some data or closes it. Am I right? But why is it closed when calling the same request, same parameters, same method with phantomjs, and it's not closed when using curl?

Hope somebody can help me to figure this out. thx!

Answer 1

Since this is a HTTPS I bet the client is closing the connection. In HTTPS this can happen when the server certificate is for example not valid. What kind of HTTPS library do you use? These libraries can be usually configured to ignore SSL CERT and continue working when it is not valid.

In curl you are actually doing that using -k (--insecure), without this it would not work. Try this without this option and if curl fails, then your server certificate is not valid. Note to get this working you usually either need to turn the checking off or to provide valid certificate to the client so it can verify it.