Showing progress in percentage while uploading and downloading using HttpWebRequest class

Question 1

You're going to need to know a few things to be successful with this.

Step 0: keep the documentation for .NET 4.0 handy:

If you look at the HttpWebRequest documentation, you'll see that GetResponse() has two similarly-named methods: BeginGetResponse() and EndGetResponse(). These methods use the oldest .NET asyncronous pattern, known as "the IAsyncResult pattern". There's thousands of pages of text about this pattern, and you can read tutorials if you want detailed information. Here's the crash course:

For a method Foo(), there is a BeginFoo() that may take parameters and must return an IAsyncResult implementation. This method performs the task Foo() performs, but does the work without blocking the current thread.
If BeginFoo() takes an AsyncCallback parameter, it expects you to provide a delegate it will call when it is finished. (This is one of several ways to know it's finished, but happens to be the technique HttpWebRequest uses.)
EndFoo() takes the IAsyncResult returned by BeginFoo() as a parameter and returns the same thing Foo() returns.

Next, there's a pitfall of which you should be aware. Controls have a property called "thread affinity", and it means it is only valid to interact with a control if you are working on the thread that created the control. This is important, because the callback method you give to BeginFoo() is not guaranteed to be called from that thread. It's actually really easy to handle this:

Every control has an InvokeRequired property that is the only safe property to call from the wrong thread. If it returns true, you know you are on an unsafe thread.
Every control has an Invoke() method that accepts a delegate parameter and will call that delegate on the thread that created the control.

With all of that out of the way, let's start looking at some code I wrote in a simple WinForms application to report the progress of downloading the Google home page. This should be similar to code that will solve your problem. It is not the best code, but it demonstrates the concepts. The form had a progress bar named progressBar1, and I called GetWebContent() from a button click.

HttpWebRequest _request;
IAsyncResult _responseAsyncResult;

private void GetWebContent() {
    _request = WebRequest.Create("http://www.google.com") as HttpWebRequest;
    _responseAsyncResult = _request.BeginGetResponse(ResponseCallback, null);           
}

This code starts the asynchronous version of GetResponse(). We need to store the request and the IAsyncResult in fields because ResponseCallback() needs to call EndGetResponse(). Everything in GetWebContent() is on the UI thread, so if you wanted to update some controls it is safe to do so here. Next, ResponseCallback():

private void ResponseCallback(object state) {
    var response = _request.EndGetResponse(_responseAsyncResult) as HttpWebResponse;
    long contentLength = response.ContentLength;
    if (contentLength == -1) {
        // You'll have to figure this one out.
    }
    Stream responseStream = response.GetResponseStream();
    GetContentWithProgressReporting(responseStream, contentLength);
    response.Close();
}

It's forced to take an object parameter by the AsyncCallback delegate signature, but I'm not using it. It calls EndGetResponse() with the IAsyncResult we got earlier, and now we can proceed as if we hadn't used asyncronous calls. But since this is an asynchronous callback, it might be executing on a worker thread, so do not update any controls directly here.

Anyway, it gets the content length from the response, which is needed if you want to calculate your download progress. Sometimes the header that provides this information isn't present, and you get -1. That means you're on your own for progress calculation and you'll have to find some other way to know the total size of the file you are downloading. In my case it was sufficient to set the variable to some value since I didn't care about the data itself.

After that, it gets the stream that represents the response and passes that along to a helper method that does the downloading and progress reporting:

private byte[] GetContentWithProgressReporting(Stream responseStream, long contentLength) {
    UpdateProgressBar(0);

    // Allocate space for the content
    var data = new byte[contentLength];
    int currentIndex = 0;
    int bytesReceived = 0;
    var buffer = new byte[256];
    do {
        bytesReceived = responseStream.Read(buffer, 0, 256);
        Array.Copy(buffer, 0, data, currentIndex, bytesReceived);
        currentIndex += bytesReceived;

        // Report percentage
        double percentage = (double)currentIndex / contentLength;
        UpdateProgressBar((int)(percentage * 100));
    } while (currentIndex < contentLength);

    UpdateProgressBar(100);
    return data;
}

This method still might be on a worker thread (since it is called by the callback) so it is still not safe to update controls from here.

The code is common for examples of downloading a file. It allocates some memory to store the file. It allocates a buffer to get file chunks from the stream. In a loop it grabs a chunk, puts the chunk in the bigger array, and calculates the progress percentage. When it's downloaded as many bytes as it expects to get, it quits. It's a bit of trouble, but if you were to use the tricks that let you download a file in one shot, you wouldn't be able to report download progress in the middle.

(One thing I feel obligated to point out: if your chunks are too small, you'll be updating the progress bar very quickly and this can still cause the form to lock up. I usually keep a Stopwatch running and try not to update progress bars more than twice a second, but there's other ways to throttle the updates.)

Anyway, the only thing left is the code that actually updates the progress bar, and there's a reason it's in its own method:

private void UpdateProgressBar(int percentage) {
    // If on a worker thread, marshal the call to the UI thread
    if (progressBar1.InvokeRequired) {
        progressBar1.Invoke(new Action<int>(UpdateProgressBar), percentage);
    } else {
        progressBar1.Value = percentage;
    }
}

If InvokeRequired returns true, it calls itself via Invoke(). If it happens to be on the correct thread, it updates the progress bar. If you happen to be using WPF, there are similar ways to marshal calls but I believe they happen via the Dispatcher object.

I know that's a lot. That's part of why the new async stuff is so great. The IAsyncResult pattern is powerful but doesn't abstract many details away from you. But even in the newer patterns, you must keep track of when it is safe to update a progress bar.

Things to consider:

This is example code. I haven't included any exception handling for brevity.
If an exception would have been thrown by GetResponse(), it will be thrown instead when you call EndGetResponse().
If an exception happens in the callback and you haven't handled it, it'll terminate its thread but not your application. This can be strange and mysterious to a newbie asynchronous programmer.
Pay attention to what I said about the content length! You may think you can ask the Stream by checking its Length property, but I've found this to be unreliable. A Stream is free to throw NotSupportedException or return a value like -1 if it's not sure, and generally with network streams if you aren't told in advance how much data to expect then you can only keep asking, "Is there more?"
I didn't say anything about uploading because:
- I'm not as familiar with uploading.
- You can follow a similar pattern: you'll probably use HttpWebRequest.BeginGetRequestStream() so you can asynchronously write the file you're uploading. All of the warnings about updating controls apply.

Question 2

As long as you set either HttpWebRequest.ContentLength or HttpWebRequest.SendChunked before calling GetRequestStream, the data you send will be sent to the server with each call to Stream.[Begin]Write. If you write the file in small chunks suggests, you can get an idea of how far along you. This is a solution for download.

Question 3

The bytesReceived variable which obtains it's value from responseStream.Read(buffer, 0, 256); returns the size of each of the single packets received, not the total thus the (do...while) will almost always return true resulting in an infinite loop, unless the downloaded content is so small that the packet combined with TCP padding ends up being larger or equal to the contentLength.

The only fix required is to have another local variable to act as an accumulator to track the total progress e.g. totalBytesReceived += bytesReceived; and use while(totalBytesReceived < contentLength) instead. I tried using currentIndex for this purpose, however it was causing issues (download stalls), so a new independent variable was used.

This was what I ended up using:

    private byte[] GetContentWithProgressReporting(Stream responseStream, long contentLength) {
UpdateProgressBar(0);

// Allocate space for the content
var data = new byte[contentLength];
int currentIndex = 0;
int bytesReceived = 0;
int totalBytesReceived = 0;
var buffer = new byte[256];
do {
    bytesReceived = responseStream.Read(buffer, 0, 256);
    Array.Copy(buffer, 0, data, currentIndex, bytesReceived);
    currentIndex += bytesReceived;
    totalBytesReceived += bytesReceived;

    // Report percentage
    double percentage = (double)currentIndex / contentLength;
    UpdateProgressBar((int)(percentage * 100));



} while (totalBytesReceived < contentLength);

//DEBUGGING:  MessageBox.Show("GetContentWithProgressReporting Method - Out of loop");
UpdateProgressBar(100);
return data;

}

Other than that the code worked really well!