I am using RestSharp as the underlying HTTP client library to make a stress/throughput test client on a black box service. Threadpool and Servicepoint connection limits have been lifted to 5000, but that shouldn't be much of a worry as we are testing around 500-1000 requests per second. A high-resolution (microsecond) timer component is used to throw out requests at the rate we want to test.
The RestSharp code roughly goes
restClient.ExecuteAsync(postRequest, res =>
{
stopwatch.Stop();
lock (this.countLocker)
{
this.roundTrips.Add(stopwatch.ElapsedMilliseconds);
if (res.ResponseStatus == ResponseStatus.Completed &&
(res.StatusCode == HttpStatusCode.OK ||
res.StatusCode == HttpStatusCode.NoContent))
{
this.responseCount++;
}
else
{
// Treat all other status codes as errors.
this.reportError(res);
}
}
});
On pumping too many requests, we'd observe the service would after some time spill some error 503 responses, but RestSharp treats those as Complete responses since that's a valid response from the server; no actual exception got thrown.
What is not clear is when RestSharp encounters exception due to underlying connection error
The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
at RestSharp.Http.GetRawResponseAsync(IAsyncResult result, Action`1 callback)
at RestSharp.Http.ResponseCallback(IAsyncResult result, Action`1 callback)
or
The underlying connection was closed: An unexpected error occurred on a receive.
at RestSharp.Http.GetRawResponseAsync(IAsyncResult result, Action`1 callback)
at RestSharp.Http.ResponseCallback(IAsyncResult result, Action`1 callback)
Which seems to suggest RestSharp is using HTTP keep-alive for connections. Is there a way to control this behaviour? I cannot seem to locate any setting to instruct RestSharp to not use keep-alive.
Beyond that, am also trying to gain better understanding on how to investigate further on the actual issue of server breaking those connections? Is it simply a matter of client issuing accumulating more connections than the server can deal with? (since it cannot keep up with its response rate)