404 - not found. But browser can read the url

Question

In my ASP.Net app, I open a request to a url as follows:

webRequest = WebRequest.Create(url);
response = webRequest.GetResponse();

But the GetResponse returns an exception of:

The remote server returned an error: (404) Not Found.

The url does in fact exist. If I copy the url and enter it in my browser, it will show the page. What could be causing this exception?

Answer 1

Depends on the URL.

Some sites are configured so that if they don't get a recognized user-agent, they'll assume the site is being crawled by an unauthorized bot and will either return a 403 Forbidden or a 404 Not Found, depending on the site.