How to read the response stream before the Http response completes

Question

When making a request using HttpWebRequest object, I need to call the method GetResponse() to send the request and get the response back.
The problem with this method is that it doesn't return the response object until all data has been received. Say I am downloading a 100 MB file, I won't be able to read it until the response finish and all the 100 MB is downloaded.
What I want is to be able to read the response stream bytes as soon as they arrive, without waiting for the response to complete.
I know I can use the Range Http header, but it won't work on my situation.

Answer 1

I think this is very close to what @Zachary suggests. And it (seems to) work(s); actually I think using using as @Zachary does is even "nicer".
My main point being I cannot see the blocking behaviour of GetResponse() you (seem to) describe.

In addition the following code only roughly shows how everything works; it will not read the stream to the end for example (unless by coincidence :)). But it should work if you copy-n-paste it into an empty "Console Application"-project in Visual Studio.

You can try using some "shorter" URL for a test. The example here starts downloading an ISO of the debian distribution (a bit more than 600 MByte). Sorry debian, I did not mean to steal your bandwidth. -> Btw: is there something sensible one can use to test such a scenario?

The Code is strongly inspired by C# - How to read a continuous stream of XML over HTTP.

namespace StreamReadWebRequest
{
    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.Net;
    using System.IO;

    class Program
    {
        static void Main(string[] args)
        {
            HttpWebRequest req;
            HttpWebResponse res = null;

            try
            {
                req = (HttpWebRequest)WebRequest.Create(
                        "http://cdimage.debian.org/debian-cd/5.0.4/i386/iso-cd/debian-504-i386-CD-1.iso");
                res = (HttpWebResponse)req.GetResponse();
                Stream stream = res.GetResponseStream();

                byte[] data = new byte[4096];
                int read;
                while ((read = stream.Read(data, 0, data.Length)) > 0)
                {
                    Process(data, read);
                }
            }
            finally
            {
                if (res != null)
                    res.Close();
            }
            Console.In.Read();
        }

        private static void Process(byte[] data, int read)
        {
            Console.Out.Write(ASCIIEncoding.ASCII.GetString(data));
        }
    }
}

Answer 2

I was looking for the same thing: server streams chunked XML data and I needed a C# client that could access this data while server is streaming. I tried many different ways to access the source (WebChannelFactory, WebClient, HttpWebRequest/Response, TcpClient) but failed so far. Finding this thread I focused on HttpWebRequest/Response where I have the same problem that following line is blocking:

HttpWebResponse resp = (HttpWebResponse)request.GetResponse();

As Artiom Chilaru stated, if it's blocking: something is wrong, because it should not. Now focusing on trying to replicate default behavior with downloading large .ISO files I found out that Fiddler was blocking the GetResponse() method!

However there is no problem to open Fiddler once the stream has been set up (i.e. GetResponse() has already been called), but during the HTTP GET if you find GetResponse() is blocking try to close Fiddler and see if your application now continuous it's normal flow (i.e. reading the stream).

Answer 3

If you set the buffer size on your read, you can read in the data in chunks... example...

 // Get the response stream
 using(Stream resStream = response.GetResponseStream())
 {

        string parseString = null;
        int    count      = 0;

        do
        {
            // Read a chunk of data
            count = resStream.Read(buf, 0, buf.Length);

            if (count != 0)
            {
                // Convert to ASCII
                parseString = Encoding.ASCII.GetString(buf, 0, count);

                // Append string to results
                sb.Append(tempString);
            }
        }
        while (count > 0);

}

Answer 4

I'm not sure what you have on your side, but I know for a fact (and I'm sure many people will agree here) that GetResponse() will NOT download the whole file back. It will send the request, wait for the response, and get the response headers.

After you have the response, you can easily get the response stream with GetResponseStream(), which is the actual data stream that's downloading from the server. And you can easily access the response stream BEFORE the whole file is downloaded. This is 100% true and tested.

If you're not getting the same behaviour (which is really strange, and shouldn't happen) could you add a code example that is not working as I explained above?

Also, do test the example posted by scherand. It just proves once again that it works just fine, without any special hacks.