.NET 4.5 HTTPWebRequest - Intermittent slowdowns with a large number of simultaneous HTTPS POST requests

Question

I have a large number (50+) of threads which make HTTPS POST requests to a web api to poll different data sets as fast as possible. It works fine half of the time, but for the other half there will be a hiccup where the bandwidth used by the application goes down significantly.

Update 1 The code listed below isnt current anymore, it was modified per Domin8urMind recommendation.

So I've looked at all the traffic in fiddler. Once a slow down starts I notice that my client establishes new HTTPS tunnels, and the slowdown ends after the new tunnels have been established. The 50+ tunnels which are established when the program first start takes only about 1 second to establish, but during a slowdown it takes aproximately 30 seconds to establish all the tunnels. This re-establishment/slowdown seems to happen like clock work every 2 minutes.

Also i've noticed that a CDN sits in between my client and the server, and that it does have DoS protection. I am unsure if this performance issue is either related to my code, or to the CDN.

So I guess my questions now would be:

• Under what conditions will a HTTPWebRequest's HTTPS connections be closed when the keep alive and pipelining features are enabled?
• Why is it taking so long for these connections to be re-established?

myMethod

private static string myMethod(string method, Dictionary<string, string> paramList = null, int maxRetryCount = 1)
{
    System.Net.ServicePointManager.Expect100Continue = false;
    System.Net.ServicePointManager.DefaultConnectionLimit = 400;
    System.Net.ServicePointManager.UseNagleAlgorithm = false;

    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("My URL");
    String postData = "My POST Data"

    request.ContentType = "application/x-www-form-urlencoded";
    request.ContentLength = postData.Length;
    request.Method = "POST";
    request.Timeout = 1000;
    request.Headers.Add("Accept-Encoding", "gzip,deflate");
    request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
    request.Proxy = null;

    try
    {
        Stream stream = request.GetRequestStream();
        stream.Write(postData, 0, postData.Length);
        stream.Close();

        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        Stream responseStream = response.GetResponseStream();
        Stream decompressedStream = null;
        if (response.ContentEncoding.ToLower().Contains("gzip"))
            decompressedStream = new GZipStream(responseStream, CompressionMode.Decompress);
        else if (response.ContentEncoding.ToLower().Contains("deflate"))
            decompressedStream = new DeflateStream(responseStream, CompressionMode.Decompress);
        else
            decompressedStream = responseStream;

        StreamReader postreqreader = new StreamReader(decompressedStream);
        var json = postreqreader.ReadToEnd();

        postreqreader.Close();
        decompressedStream.Close();
        responseStream.Close();

        return json;
    }
    catch (Exception e)
    {
        if(e.Message != "The operation has timed out")
            Console.Write("\nGot exception " + e.Message + " " + e.StackTrace + "\n\n\n");
    }
    return null;
}

}

Exception 1

Got exception The request was aborted: The request was canceled.    at System.Ne
t.GZipWrapperStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.StreamReader.ReadBuffer()
   at System.IO.StreamReader.ReadToEnd()
   at myMethod in myFile:line 337

Line 337 is "var json = postreqreader.ReadToEnd();"

Exception 2

Got exception Safe handle has been closed    at System.Runtime.InteropServices.S
afeHandle.DangerousAddRef(Boolean& success)
   at System.StubHelpers.StubHelpers.SafeHandleAddRef(SafeHandle pHandle, Boolea
n& success)
   at System.Net.UnsafeNclNativeMethods.OSSOCK.setsockopt(SafeCloseSocket socket
Handle, SocketOptionLevel optionLevel, SocketOptionName optionName, Int32& optio
nValue, Int32 optionLength)
   at System.Net.Sockets.Socket.SetSocketOption(SocketOptionLevel optionLevel, S
ocketOptionName optionName, Int32 optionValue, Boolean silent)
   at System.Net.Sockets.NetworkStream.SetSocketTimeoutOption(SocketShutdown mod
e, Int32 timeout, Boolean silent)
   at System.Net.ConnectStream.DrainSocket()
   at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean abort
ing)
   at System.Net.ConnectStream.System.Net.ICloseEx.CloseEx(CloseExState closeSta
te)
   at System.Net.HttpWebRequest.SetAndOrProcessResponse(Object responseOrExcepti
on)
   at System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult retu
rnResult)
   at System.Net.Connection.ReadComplete(Int32 bytesRead, WebExceptionStatus err
orStatus)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetriev
edStream, Boolean probeRead)
   at System.Net.ConnectStream.ProcessWriteCallDone(ConnectionReturnResult retur
nResult)
   at System.Net.HttpWebRequest.CheckDeferredCallDone(ConnectStream stream)
   at System.Net.HttpWebRequest.GetResponse()
   at myMethod in myFile:line 325

Line 325 is "HttpWebResponse response = (HttpWebResponse)request.GetResponse();"

Answer 1

I recommend you change your code using the "using" clause which will close connections correctly, something like:

HttpWebRequest httpWebRequest = (HttpWebRequest)HttpWebRequest.Create("My URL");

using (HttpWebResponse httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse())
{
  using (Stream stream = httpWebResponse.GetResponseStream())
  {
    using (StreamReader reader = new StreamReader(stream))
    {
      ...
    }
  }
}

The problem you are having with your existing loop is that when you catch an exception, the connection is not disposed of correctly and you would need to wait for the Garbage Collector to cleanup these connections before being able to create another.

I suspect that is your delay. You can verify this by watching the .net GC counter in perfmon and if '% in GC' is spiking during these "slow downs", then you have your culprit .