Question

I'm trying to get mechanize to go over a SSH tunnel SOCKS5 proxy. Some reason it keeps getting an error during the HTTP requests. Like it's getting an EOF before it should. If I turn off the proxy setting it works, I'm not sure why it would be receiving multiple connection requests from the proxy.

What might cause multiple requests to be sent back? No other traffic is going over the tunnel.

require 'rubygems'
require 'mechanize'
require 'logger'
require 'pp'

a = Mechanize.new { |agent|
    agent.user_agent_alias = 'Windows Mozilla'
    agent.log = Logger.new $stdout
    agent.agent.http.debug_output = $stdout
    agent.set_proxy 'localhost', 9999
}

page = a.get('http://www.whatsmyip.org')
PP.pp(page)

Output:

I, [2013-07-19T00:18:35.858550 #22936]  INFO -- : Net::HTTP::Get: /
D, [2013-07-19T00:18:35.858650 #22936] DEBUG -- : request-header: accept-encoding => gzip,deflate,identity
D, [2013-07-19T00:18:35.858676 #22936] DEBUG -- : request-header: accept => */*
D, [2013-07-19T00:18:35.858699 #22936] DEBUG -- : request-header: user-agent => Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6
D, [2013-07-19T00:18:35.858778 #22936] DEBUG -- : request-header: accept-charset => ISO-8859-1,utf-8;q=0.7,*;q=0.7
D, [2013-07-19T00:18:35.858801 #22936] DEBUG -- : request-header: accept-language => en-us,en;q=0.5
D, [2013-07-19T00:18:35.858826 #22936] DEBUG -- : request-header: host => www.whatsmyip.org
opening connection to localhost:9999...
opened
<- "GET http://www.whatsmyip.org/ HTTP/1.1\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept: */*\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: www.whatsmyip.org\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error end of file reached, and retry
opening connection to localhost:9999...
opened
<- "GET http://www.whatsmyip.org/ HTTP/1.1\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept: */*\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: www.whatsmyip.org\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error end of file reached
Conn close because of error end of file reached
opening connection to localhost:9999...
opened
<- "GET http://www.whatsmyip.org/ HTTP/1.1\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept: */*\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: www.whatsmyip.org\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error end of file reached, and retry
opening connection to localhost:9999...
opened
<- "GET http://www.whatsmyip.org/ HTTP/1.1\r\nAccept-Encoding: gzip,deflate,identity\r\nAccept: */*\r\nUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Language: en-us,en;q=0.5\r\nHost: www.whatsmyip.org\r\nConnection: keep-alive\r\nKeep-Alive: 300\r\n\r\n"
Conn close because of error end of file reached
Conn close because of error end of file reached
/home/krizzo/.rvm/gems/ruby-2.0.0-p0/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:959:in `rescue in request': too many connection resets (due to end of file reached - EOFError) after 0 requests on 11809620, last used 1374214715.8607903 seconds ago (Net::HTTP::Persistent::Error)
    from /home/krizzo/.rvm/gems/ruby-2.0.0-p0/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:968:in `request'
    from /home/krizzo/.rvm/gems/ruby-2.0.0-p0/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:257:in `fetch'
    from /home/krizzo/.rvm/gems/ruby-2.0.0-p0/gems/mechanize-2.7.1/lib/mechanize.rb:431:in `get'
    from ./imageDump.rb:93:in `<main>'
Was it helpful?

Solution

You should be using a http proxy.

If you really want to use a socks5 proxy you could look into socksify.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top