C# HttpWebRequest to Socket 3 second delay in writing to Request Stream

Question

I have a set up with a custom HTTP server using sockets, and a client that uses HttpWebRequest to connect to that server. The client is multi-threaded, using this code:

ServicePointManager.UseNagleAlgorithm = false;
ServicePointManager.Expect100Continue = false;
ServicePointManager.DefaultConnectionLimit = 48;

var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = verb;
request.Timeout = timeout;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Proxy = null;
request.SendChunked = false;

/*offending lines*/
var writer = new StreamWriter(request.GetRequestStream(),Encoding.UTF8);
writer.Write(stringData);
writer.Close();
/*end offending lines*/

var response = (HttpWebResponse)request.GetResponse();

I have been running thousands of requests through this code, and it works very well most of the time. The average response time is about 200ms, including some work done on the server.

The problem is that about 0.2% of the requests are very slow. When measuring I notice that the slow requests are pausing for 3 seconds (+- 20ms) when getting and writing to the request stream. GetResponse() is still fast, and nothing on the server takes extra time.

The server is implemented as a listener Socket listening asyncronously on a port, using Listen() and BeginAccept(). After a connection is accepted, BeginRecieve is called on the new Socket (aquired from EndAccept()) and BeginAccept is called again on the listener socket. From what I've read, this is the way to do it.

So my question is: what happens during GetRequestStream() and writing to it? Is data actually being sent here or is it flushed on GetResponse()? How come some of the requests take 3 seconds longer without any apparent reason? Is my server implementation flawed in some way? The weird thing is that it is always 3 seconds, except sometimes it can be 9 seconds. It seems to be a multiple of 3, for some reason.

A few possible scenarios:

• The connection is initiated on GetRequestStream() and the server does not accept it right away. How can I test if this is the case?
• Something is going on behind the scenes on GetRequestStream() or writing to it.
• Waiting for network hardware?
• Waiting for DNS resolution?
• Something else?

Any help is very appreciated.

EDIT: I've tried connecting through a web proxy, and then the delay moves from GetRequestStream() to GetResponse(), indicating that the problem indeed lies in the web server. It seems like it is not accepting the connection. I realize that there might be some delay between EndAccept() and BeginAccept(), but it should not amount to a 3 second delay, right?

I've also tried running my client single-threaded, and the problem seems to disappear. This suggests that the connection delay only occurs when multiple requests are made at the same time.

Answer 1

Preliminary testing suggests that the issue was indeed connection crowding on the server. With some restructuring of Socket code, I managed to get rid of the slow connections. I will run more testing tonight to make sure, but it looks promising!

Answer 2

I guess I'm a bit too late, but I've encountered similar problem and found and article, which describes what seems to be pretty low-level reason for such an issue - http://www.percona.com/blog/2011/04/19/mysql-connection-timeouts/

Answer 3

Maybe its because you don't dispose the request stream. Try to replace your offending lines with the following:

var stream = request.GetRequestStream();
var buffer = Encoding.UTF8.GetBytes(stringData);
stream.Write(buffer, 0, buffer.Length);
stream.Dispose();