c# - httpwebrequest status code returns 200 instead of 404

Question

I'm trying this code..

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.goo4le.com/");
request.Method = "HEAD";
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
  Console.Write((int)response.StatusCode);
}

goo4le is a non existing domain. So its supposed to return 404. Instead it returns 200 status.

I think its because of my broadband provider using a custom 404 page.

This is what i see when i enter goo4le.com in my browser.

Can someone tell me how to get the real http status instead of my browser status?

Answer 1

I actually don't get any status code when running this, I get a DNS error saying I cant lookup the domain.

I imagine you are exactly right about the ISP, they may be doing this via a DNS redirection given you dont get this error. You could solve this by using a DNS server other than the one your ISP provides, try googles 8.8.8.8, 8.8.4.4 (https://developers.google.com/speed/public-dns/)

This from their FAQs

How is Google Public DNS different from my ISP's DNS service or other open DNS resolvers? How can I tell if it is better?

Open resolvers and your ISP all offer DNS resolution services. We invite you to try Google Public DNS as your primary or secondary DNS resolver along with any other alternate DNS services. There are many things to consider when identifying a DNS resolver that works for you, such as speed, reliability, security, and validity of responses. Unlike Google Public DNS, some ISPs and open resolvers block, filter, or redirect DNS responses.

How does Google Public DNS handle non-existent domains?

If you issue a query for a domain name that does not exist, Google Public DNS always returns an NXDOMAIN record, as per the DNS protocol standards. The browser should show this response as a DNS error. If, instead, you receive any response other than an error message (for example, you are redirected to another page), this could be the result of the following: A client-side application such as a browser plug-in is displaying an alternate page for a non-existent domain. Some ISPs may intercept and replace all NXDOMAIN responses with responses that lead to their own servers. If you are concerned that your ISP is intercepting Google Public DNS requests or responses, you should contact your ISP.

Answer 2

You can try to disable redirecting, unfortunatelly i can't test it, since i got another provider (if that's the problem).

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.goo4le.com/");
request.Method = "HEAD";
request.AllowAutoRedirect = false;
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
  Console.Write((int)response.StatusCode);
}

if it is working, it will most likely throw an exception, since everything that returns a statuscode that's not 200 throws one...... if you wanna catch it try this:

try
{
    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://www.goo4le.com/");
    request.Method = "HEAD";
    request.AllowAutoRedirect = false;
    using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
  Console.Write((int)response.StatusCode);
}

}
catch (WebException e)
{
   // in this case it was a status code exception (not status 200...)
   if (e.Response != null) Console.Write((int)e.Response.StatusCode);
   else throw;
}