System.Net.WebRequest not respecting hosts file

Question

Is there a way to get a System.Net.WebRequest or System.Net.WebClient to respect the hosts or lmhosts file?

For example: in my hosts file I have:

10.0.0.1  www.bing.com

When I try to load Bing in a browser (both IE and FF) it fails to load as expected.

Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1 
WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected)
WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds

Similarly:

WebClient wc = new WebClient();
wc.DownloadString("http://www.bing.com"); //succeeds 

Why would System.Net.Dns respect the hosts file but System.Net.WebRequest ignore it? What do I need to change to make the WebRequest respect the hosts file?

Additional Info:

• If I disable IPv6 and set my IPv4 DNS Server to 127.0.0.1, the above code works (fails) as expected. However if I add my normal DNS servers back as alternates, the unexpected behavior resumes.
• I've reproduced this on 3 Win7 and 2 Vista boxes. The only constant is my company's network.
• I'm using .NET 3.5 SP1 and VS2008

Edit

Per @Richard Beier's suggestion, I tried out System.Net tracing. With tracing ON the WebRequest fails as it should. However as soon as I turn tracing OFF the behavior reverts to the unexpected success. I have reproduced this on the same machines as before in both debug and release mode.

Edit 2

This turned out to be the company proxy giving us issues. Our solution was a custom proxy config script for our test machines that had "bing.com" point to DIRECT instead of the default proxy.

Answer 1

I think that @Hans Passant has spotted the issue here. It looks like you have a proxy setup in IE.

Dns.GetHostAddresses("www.bing.com")[0]; // 10.0.0.1  

This works because you are asking the OS to get the IP addresses for www.bing.com

WebRequest.Create("http://www.bing.com/").GetResponse(); // unexpectedly succeeds

This works because because you are asking the framework to fetch a path from a server name. The framework uses the same engine and settings that IE frontend uses and hence if your company has specified by a GPO that you use a company proxy server, it is that proxy server that resolves the IP address for www.bing.com rather than you.

WebRequest.Create("http://10.0.0.1").GetResponse(); // throws exception (expected) 

This works/fails because you have asked the for the framework to fetch you a webpage from a specific server (by IP). Even if you do have a proxy set, this proxy will still not beable to connect to this IP address.

I hope that this helps.

Jonathan

Answer 2

I'm using VS 2010 on Windows 7, and I can't reproduce this. I made the same hosts-file change and ran the following code:

Console.WriteLine(Dns.GetHostAddresses("www.bing.com")[0]);             // 10.0.0.1     
var response = WebRequest.Create("http://www.bing.com/").GetResponse(); // * * *
Console.WriteLine(new StreamReader(response.GetResponseStream()).ReadToEnd());

I got an exception on the line marked "* * *". Here's the exception detail:

System.Net.WebException was unhandled
  Message=Unable to connect to the remote server
  Source=System
  StackTrace:
       at System.Net.HttpWebRequest.GetResponse()
       at ConsoleApplication2.Program.Main(String[] args) in c:\Data\Projects\ConsoleApplication2\ConsoleApplication2\Program.cs:line 17
  InnerException: System.Net.Sockets.SocketException
       Message=A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.1:80
       Source=System
       ErrorCode=10060

Maybe it's an issue with an earlier .NET version, that's now fixed in .NET 4 / VS 2010? Which version of .NET are you using?

I also found this thread from 2007, where someone else ran into the same problem. There are some good suggestions there, including the following:

• Turn on system.net tracing

• Work around the problem by using Dns.GetHostAddresses() to resolve it to an IP. Then put the IP in the URL - e.g. "http://10.0.0.1/". That may not be an option for you though.

In the above thread, mariyaatanasova_msft also says: "HttpWebRequest uses Dns.GetHostEntry to resolve the host, so you may get a different result from Dns.GetHostAddresses".

Answer 3

This stackoverflow answer has the quick-and dirty configs to disable your proxy within .NET. Put it in your web.config or app.config and then System.Web won't look to the underlying system to get its proxy config.

Answer 4

You should overwrite the default proxy. HttpWebRequest & WebRequest will set a default proxy if present in Internet Explorer and your file hosts will be bypassed.

request.Proxy = new WebProxy();

The following is just an example of code:

try
{
   HttpWebRequest request = (HttpWebRequest)WebRequest.Create("www.bing.com");
   request.Proxy = new WebProxy();
   request.Method = "POST";            
   request.AllowAutoRedirect = false;

   HttpWebResponse response = (HttpWebResponse)request.GetResponse();               
   if (response.StatusCode == HttpStatusCode.OK)
   {
      //some code here
   }
}
catch (exception e)
{
   //Some other code here
}