Timeout with HttpWebResponse in .NET

Question

I have the following code, and after ~60 times calling it (20 concurrent connections) it starts timing out. if i lower the timeout from 10 minutes to 1 minute, they start timing out at ~34 downloads. what gives? i know that you can get this if you don't properly close your response, but i'm definitely closing it:

    //===============================================================================
    /// <summary>
    /// Executes the request and returns the response as a byte array. Useful if the 
    /// response should return a file.
    /// </summary>
    private static byte[] GetResponse(HttpWebRequest webRequest)
    {
        //---- declare vars
        HttpWebResponse response = null;
        List<byte> buffer = new List<byte>();
        int readByte;

        //---- try to get the response, always wrap it.
        try
        { response = webRequest.GetResponse() as HttpWebResponse; }
        //---- catch all
        catch (Exception e)
        {
            if (response != null) { response.Close(); }
            throw new ConnectionFailedException("Failed to get a response", e);
        }

        try
        {
            //---- if the response is ok
            if (response.StatusCode == HttpStatusCode.OK)
            {
                //---- get the response stream
                using (Stream stream = response.GetResponseStream())
                {
                    //---- read each byte, one by one into the byte buffer
                    while ((readByte = stream.ReadByte()) > -1)
                    {
                        buffer.Add((byte)readByte);
                    }
                    //---- close the stream
                    stream.Close();
                    response.Close();
                }

                //---- return the buffer as a byte array
                return buffer.ToArray();
            }
            //---- if the request wasn't auth'd
            else if (response.StatusCode == HttpStatusCode.Forbidden || response.StatusCode == HttpStatusCode.Unauthorized)
            {
                if (response != null) { response.Close(); }
                throw new AuthenticationFailedException(response.StatusDescription);
            }
            //---- any other errors
            else
            {
                if (response != null) { response.Close(); }
                throw new ConnectionFailedException(response.StatusDescription);
            }
        }
        finally { if (response != null) { response.Close(); } }
    }
    //===============================================================================

thoughts?

also, i'm creating it with both the TimeOut and ReadWriteTimeout set to 10 minutes:

//---- create the web request HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest;

//---- set a 10 minute timeout webRequest.Timeout = 600000; webRequest.ReadWriteTimeout = 600000;

Answer 1

System.Net.ServicePointManager.DefaultConnectionLimit = 200;

^^ done.

that was it.

Answer 2

How about simplifying your code a bit:

using (var client = new WebClient())
{
    byte[] result = client.DownloadData("http://example.com");
}

Answer 3

Set KeepAlive property to false by:

webRequest.KeepAlive = false;

Release the resource in the finally statement.

Answer 4

Not tested, but a bit cleaner.

private static byte[] GetResponse(HttpWebRequest webRequest)
{
        using (var response = (HttpWebResponse)webRequest.GetResponse())
        {
            switch (response.StatusCode)
            {
                case HttpStatusCode.Forbidden:
                case HttpStatusCode.Unauthorized:
                    throw new AuthenticationFailedException(response.StatusDescription);
                    break;
                case HttpStatusCode.OK:
                    break; // to get through
                default:
                    throw new ConnectionFailedException(response.StatusDescription);
            }

            using (Stream stream = response.GetResponseStream())
            {
                // you should really create a large buffer and read chunks.
                var buffer = new byte[response.ContentLength];
                var bytesRead = 0;
                while (bytesRead < buffer.Length)
                {
                   var bytes = stream.Read(buffer, bytesRead, buffer.Length - bytesRead);
                   bytesRead += bytes;
                }

                return buffer;
            }

        }
}

Edit:

Changed so that the buffer allocation uses ContentLength. It's always exact unless chunked encoding is used.