Download content from the internet with code

Question

I have to download some content from a website every day so I figure it will be nice to have a program that will do it... The problem is that the website requires authentication.

My current solution is by using System.Windows.Forms.WebBrowser control. I currently do something like:

/* Create browser */
System.Windows.Forms.WebBrowser browser = new System.Windows.Forms.WebBrowser();

/* navigate to desired site */ 
browser.Navigate("http://stackoverflow.com/");

// wait for browser to download dom

/* Get all tags of type input */
var elements = browser.Document.Body.GetElementsByTagName("input");

/* let's look for the one we are interested */
foreach (System.Windows.Forms.HtmlElement curInput in elements)
{
       if (curInput.GetAttribute("name") == "q") // 
       {
             curInput.SetAttribute("value", "I changed the value of this input");
             break;
       }
}

// etc

I think this approach works but is not the best solution. I have tried to use the webclient class and that seems to work but for some reason it does not work. I belive the reason why it does not work is because I have to save the cookies?

So my question is how will I be able to track all the bytes that get send to the server and all the bytes that get responded in order to download what I need. In other words I will like to have the webclient act as a webrowser and once I get to the part I need by just looking at the source I should be able to parser the data that I need.

I will appreciate if someone can show me an example of how to do so. Google chrome does a pretty good job displaying lots of information:

Thanks in advance,

Antonio

Answer 1

Answering your question:

1. The best utility i know to track traffic is Fiddler (its free).
2. For sending advanced HTTP requests, you should use class System.Net.HttpWebRequest, which also has property CookieContainer, and Headers, allowing you to do what ever you want.

Hope it helps.