Anyone have sample code for doing a “chunked” HTTP streaming download of one web directly to a upload to a separate web server?

Question

Background - I'm trying to stream an existing webpage to a separate web application, using HttpWebRequest/HttpWebResponse in C#. One issue I'm striking is that I'm trying to set the file upload request content-length using the file download's content-length, HOWEVER the issue seems to be when the source webpage is on a webserver for which the HttpWebResponse doesn't provide a content length.

HttpWebRequest downloadRequest = WebRequest.Create(new Uri("downloaduri")) as HttpWebRequest;
 using (HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
 {
   var uploadRequest = (HttpWebRequest) WebRequest.Create(new Uri("uripath"));
   uploadRequest.Method = "POST";
   uploadRequest.ContentLength = downloadResponse.ContentLength;  // ####

QUESTION : How could I update this approach to cater for this case (when the download response doesn't have a content-length set). Would it be to somehow use a MemoryStream perhaps? Any sample code would be appreciated. In particular is there a code sample someone would have that shows how to do a "chunked" HTTP download & upload to avoid any issues of the source web server not providing content-length?

Thanks

Answer 1

As I already applied in the Microsoft Forums, there are a couple of options that you have.

However, this is how I would do it with a MemoryStream:

HttpWebRequest downloadRequest = WebRequest.Create(new Uri("downloaduri")) as HttpWebRequest;

byte [] buffer = new byte[4096];
using (MemoryStream ms = new MemoryStream())
using (HttpWebResponse downloadResponse = downloadRequest.GetResponse() as HttpWebResponse)
{
    Stream respStream = downloadResponse.GetResponseStream();
    int read = respStream.Read(buffer, 0, buffer.Length);

    while(read > 0)
    {
        ms.Write(buffer, 0, read);
        read = respStream.Read(buffer, 0, buffer.Length);
    }

    // get the data of the stream
    byte [] uploadData = ms.ToArray();

    var uploadRequest = (HttpWebRequest) WebRequest.Create(new Uri("uripath"));
    uploadRequest.Method = "POST";
    uploadRequest.ContentLength = uploadData.Length;

    // you know what to do after this....
}

Also, note that you really don't need to worry about knowing the value for ContentLength a priori. As you have guessed, you could have set SendChunked to true on uploadRequest, and then just copied from the download stream into the upload stream. Or, you can just do the copy without setting chunked, and HttpWebRequest (as far as I know) will buffer the data internally (make sure AllowWriteStreamBuffering is set to true on uploadrequest) and figure out the content length and send the request.