HTTPWebRequest.GetResponse() failing with authenticated requests through a transparent proxy

Question

We're using the HTTPWebRequest objects to make HTTP requests to our application and we're having a problem when the request requires authentication and there is a transparent proxy (Squid 3.1.10).

string url = "http://www.icode.co.uk/test/auth.php";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Credentials = new NetworkCredential("username", "password");

HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);

MessageBox.Show(reader.ReadToEnd());

reader.Close();
stream.Close();
response.Close();

Our original code used the WebClient class which exhibited the same problem.

The first time this code runs, it displays the result correctly. When the code runs a second time, it fails on the GetResponse() line with:

System.Net.WebException was unhandled
  Message="The server committed a protocol violation. Section=ResponseStatusLine"
  Source="System"
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at Dummy.DummyForm.button1_Click(Object sender, EventArgs e) in H:\Trial\Dummy\DummyForm.cs:line 42
       at ...

On Windows 7, restarting the process causes it to recover and work once, but Server 2003 requires a full reboot.

Looking at the network capture, two requests are identical to start with, the initial unauthenticated request is sent and the server replies, but the failing requests sends the 2nd authenticated request in the middle of the initial reply as if it's ignoring the Content-Length header (which is correct). It then receives the rest of the initial reply and fails with the protocol error.

It does seem odd that the client (HTTPWebRequest) doesn't close the connection cleanly though.

When the proxy is not in use (non port 80 or internal traffic) the requests all work as expected. When there is no authentication, it also works as it only makes the single request.

I've already reduced the problem code to the minimum and reproduced it with the MSDN sample, but does anyone know if this is a known issue or a problem in our (.NET or Squid) configuration?

Answer 1

Since it only fails the second time, would

request.KeepAlive = false;

make a difference?

Answer 2

I think NTLM authentication (NetworkCredential) does not work at the same time with transparent proxy feature of SQUID. :-(

http://www.squid-cache.org/mail-archive/squid-users/201110/0025.html

Could you try another authentication scheme?

Answer 3

Try authenticating yourself, with

request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(username + ":" + password));

before the request.GetResponse();

This worked for me. First I tried putting in the whole string myself, which didn't work!