HttpWebRequest.GetResponse Operation has timed out

Question

I'm trying to get simple gzip encoded html response from a website and it keeps getting time out, following is my code:

HttpWebRequest httpClient = (HttpWebRequest)WebRequest.Create(url);
httpClient.Method = "GET";
httpClient.Accept = "text/html, application/xhtml+xml, */*";
httpClient.Headers.Add("Accept-Encoding: gzip, deflate");
httpClient.Headers.Add("Accept-Language: en-US");
httpClient.Headers.Add("DNT: 1");
httpClient.ProtocolVersion = HttpVersion.Version10;
httpClient.KeepAlive = true;
httpClient.Timeout = System.Threading.Timeout.Infinite;
httpClient.CookieContainer = cookieJar;

String responseAsText;
using (HttpWebResponse response = (HttpWebResponse)httpClient.GetResponse())
{
     System.IO.StreamReader sr;
     if (response.ContentEncoding.Equals("gzip"))
     {
          sr = new StreamReader(new GZipStream(response.GetResponseStream(), CompressionMode.Decompress));
     }
     else
     {
           sr = new System.IO.StreamReader(response.GetResponseStream());
     }
     responseAsText = sr.ReadToEnd();
 }

The url I'm trying to hit is "https client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx"

This works perfectly fine in the Browser, using Fiddler I viewed the browser's Request header and since its Transfer-Encoding: chunked, I have used HttpVersion10

I have also tried setting httpClient.Timeout = System.Threading.Timeout.Infinite, but it never gets back with a response, however in browser the response gets in few seconds.

Please someone help me in achieving this.

Answer 1

probably you can try setting Agent property, so it doesn't recognize you as a bot.

Answer 2

I think Nero has answered your question ..

Try adding these Lines in your code..

request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";