HTTPS Proxy and LWP::UserAgent

Question

I have read a number of threads on a number of sites and am still unable to make this work.

I have a client machine (OSX) with OpenSSL 0.9.8r running perl 5.12.4, with LWP 6.0.4, updated Crypt::SSLeay, Net::SSL etc. I am trying to connect to an HTTPS site (https://github.com in the example) via a WinGate proxy that I have running on a Windows VM. Note that my actual application is attaching to an SSL webservice that I have no control over.

From Firefox, pointed to the proxy everything is copacetic. The page loads successfully and I see the connections in the Proxy software Activity monitor. I'll be darned if I can make it work in Perl though. I've started with the code from this Stack Overflow Question : How do I force LWP to use Crypt::SSLeay for HTTPS requests? And added some debugging and additional output. Here's were I stand now:

#!/usr/bin/perl

use strict;
use warnings;
use Net::SSL (); # From Crypt-SSLeay

BEGIN {
  $Net::HTTPS::HTTPS_SSL_SOCKET_CLASS = "Net::SSL"; # Force use of Net::SSL
  $ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
#  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

use LWP::UserAgent;
my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);

print "--\n";
print "$_\n" for grep { $_ =~ /SSL/ } keys %INC;
print "--\n";

if ($response->is_success) {
     print $response->decoded_content;  # or whatever
     exit(0);
}
else {
 print "\nFail:\n";
     print $response->status_line ."\n";
     exit(1);
}

Here's the output from this code:

--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
--

Fail:
500 Can't connect to github.com:443 (Crypt-SSLeay can't verify hostnames)

If I then uncomment $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;, I do see a single connect to github.com:443 on the proxy and then nothing. (Note it works great from a web browser through the proxy). After much hanging I get the following output from the script:

SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
--
Crypt/SSLeay.pm
Crypt/SSLeay/X509.pm
Net/SSL.pm
Crypt/SSLeay/CTX.pm
Crypt/SSLeay/MainContext.pm
--

Fail:
500 SSL negotiation failed: 

If anyone can provide some direction here I would greatly appreciate it!

Answer 1

I just uploaded the LWP::Protocol::connect module to CPAN. This module adds the missing HTTP/CONNECT method support to LWP.

  use LWP::UserAgent;

  $ua = LWP::UserAgent->new(); 
  $ua->proxy('https', 'connect://proxyhost.domain:3128/');

  $ua->get('https://www.somesslsite.com');

With this module you can use the regular IO::Socket::SSL implementation for LWP >=6.00.

Answer 2

Why would you want "Force use of Net::SSL". Try

#!/usr/bin/perl    
use strict;
use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://192.168.1.11:80';
#  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

my $ua = LWP::UserAgent->new();
my $req = HTTP::Request->new('GET','https://github.com/');
my $response = $ua->request($req);
print $response->code ."\n";

An out put of 200 should mean that there were no errors.

A below sample code of mine works perfectly

#!/usr/bin/perl
use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080'; #Valid HTTPS proxy taken from http://hidemyass.com/proxy-list/
  $ENV{HTTPS_DEBUG} = 1;
}

my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://www.nodeworks.com');
my $res = $ua->request($req);
print $res->code, "\n";

Output-

200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A

Tool completed successfully

With https://github.com/ the output is-

200
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:SSLv3 read finished A

Tool completed successfully

So having said all this. Your code version (below) should work fine-

use warnings;
use LWP::UserAgent;

BEGIN {
  $ENV{HTTPS_PROXY} = 'https://176.9.209.113:8080';
  $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; #works even with this
  $ENV{HTTPS_DEBUG} = 1;  #Add debug output
}

my $ua = new LWP::UserAgent;
my $req = new HTTP::Request('GET', 'https://github.com/');
my $res = $ua->request($req);
print $res->code, "\n";

if ($res->is_success) {
     print $res->decoded_content;  # or whatever
     exit(0);
}
else {
 print "\nFail:\n";
     print $res->status_line ."\n";
     exit(1);
}

Answer 3

I pretty much ran into the same problem. Here are the things that fixed it for me:

• make sure you use Crypt::SSLeay 0.58 or lower. At least 0.64 blocked endlessly for me (as described at http://cpanforum.com/posts/13808 ), while 0.57/0.58 worked.
• make sure Net::HTTP has been updated to at least 6.03 (to avoid https://rt.cpan.org/Public/Bug/Display.html?id=72790 )

Answer 4

Instead of using Net::SSL which does not provide much host verification (and no SNI) you can use Net::SSLGlue::LWP. This monkey-patches LWP so that https_proxy can be used with the default SSL backend IO::Socket::SSL:

use Net::SSLGlue::LWP; # do this first
use LWP::Simple;
... continue with normal LWP stuff..

Answer 5

I know this may be a dead question, but if anyone else hits it I've another angle... I can't promise any answers, but we've faced a long-standing problem at $work in this area, but with the Squid proxy, and maybe specific to use of X509 client certs.

The use of the Net::SSL override is part of the solution, but I would fear that WinGate could be the problem (and not something I can help with) although in our case we contact the proxy over http (not sure how LWP deals with proxy+https).

For the record, here's an example of the precise form of code we use:

use Net::SSL;
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS}="Net::SSL";
use LWP::UserAgent;
use LWP::Protocol::https;
my $ua = LWP::UserAgent->new;
$ENV{HTTPS_PROXY} = 'http://cache.local.employer.co.uk:80';
$ua->get("https://example.com/");

This is Perl 5.8.8 with recent CPAN installs (hence separation of L:P:https), so we have a fresh Net::HTTP.

I was going to mention some versions of Net::HTTP are crocked, but I just realised that was my CPAN bug in Martin's reply :)

Sorry if this doesn't add anything.

Answer 6

I've sent a pull-request on libwww-perl repository to fix (or may be workaround ...) the issue.

The comment of this PR shows a simple program that connects with https to github.com through a proxy. With this patch there's no need to mess around with %ENV in your program.

Another advantage is that you can re-use the usual https_proxy setup.

Answer 7

There has been a error in earl 5.8 and some further module where environment variable HTTP_PROXY does not appropriately sets the proxy connection.

Your case has a issue where the bug reported is as mentioned here https://bugzilla.redhat.com/show_bug.cgi?id=1094440

Better way to use this is without environment variable and use LWP UserAgent

 `use LWP::UserAgent;
  $ua = LWP::UserAgent->new(); 
  $ua->proxy('https', 'connect://proxyhost.domain:3128/');`

Answer 8

#!/usr/bin/env perl 
# 
# mimvp.com
# 2017-03-28

use CGI;
use strict;
use LWP::UserAgent;


our %proxy_https = ("https", "connect://173.233.55.118:443");
our $mimvp_url = "https://proxy.mimvp.com/exist.php";

## https
## 1. download LWP-Protocol-connect (wget http://search.cpan.org/CPAN/authors/id/B/BE/BENNING/LWP-Protocol-connect-6.09.tar.gz)
## 2. tar zxvf LWP-Protocol-connect-6.09.tar.gz 
##    cd LWP-Protocol-connect-6.09
##    perl Makefile.PL
##    make
##    sudo make install
sub test_connect {
	my ($url, %proxy) = @_;
	
	print "proxy  : $proxy{'http'}\n";
	print "https  : $proxy{'https'}\n";
	print "socks4 : $proxy{'socks4'}\n";
	print "socks5 : $proxy{'socks5'}\n";
	print "url : $url\n";
	
	my $browser = LWP::UserAgent->new();
	$browser->env_proxy();
	
# 	# 设置的代理格式
	$browser->proxy(%proxy);
	$browser->timeout(30);
	$browser->agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36');
	
# 	my $req = new HTTP::Request('GET', $url);
# 	my $response = $browser->request($req);
	my $response = $browser->get($url);  				# 爬取的网址
	my $is_success = $response->is_success();			# 1
	my $content_type = $response->content_type();		# text/html
	my $content = $response->content();					# 网页正文
	my $content_length = length($content);				# 网页正文长度
	
	print "$is_success\n";
	print "$content_type\n";
	print "$content_length\n";
	print "$content\n";
}

test_connect($mimvp_url, %proxy_https);		# https

## perl mimvp-proxy-perl.pl