IIS performance bottleneck with ASP.NET HttpHandler and WCF Client in the Website

Question

I have a Performance Problem with our ASP.NET web application and don't know where to investigate. When I enable the WCF Client in our web application, the web performance decreases and incoming web request start being queued.

Situation:

ASP.NET application (IHttpAsyncHandler) as a notification channel: The handler keeps the client connection open for about 20 seconds (for direct push Messages). After 20 seconds, the HTTP request is completed and closed and the client will reconnect after a few seconds. -->This all runs well with about 500 concurrent clients. There are about 20 to 50 (re-)connects a second.

To do logging, we have a trace listener in the web application, which sends the log messages to a WCF Service, using TCP.NET binding. Each connect/disconnect in the IHttpAsyncHandler is sent to the trace listener.

Problem:

Without trace listener everything works well. When I enable the trace listener, the Performance of the web application decreases dramatically. The web requests start being queued. After a few seconds, the request queue grows to several hundred entries. The web starts rejecting web connections.

I wonder why the enabling of the Trace Listener/WCF Client causes so much performance impact. I'm not sure if this is more a issue with Server/WCF throttling or it has to do with performance limitations. Could this some kind of connection pool limitation issue? Which performance counters would help me to identify the cause of this issue?

I would appreciate any ideas.

Server A (Http Handler): Win 2003 R2 SP2, IIS 6
Server B (WCF Host): Win 2008 R2 SP1, IIS 7, WAS hosting

---

Edit: The WCF roxy is created and closed for every message received in the TraceListener.
Both bindings, WCF Client and Server have a maxConnetions=300 configured.

Answer 1

It turned out that this is all related to the server performance. The Server is running close to CPU maximum. When enabling the trace listener, each call takes about 50 miliseconds more. Our webserver is then not capable to process all requests in time and starts queueing them.

Answer 2

Are you keeping the wcf client open too long or not closing it properly, why are you using wcf as a trace listener in a this scenario with many concurrent users, doesn't seem like the optimal solution.

Also you can try increasing the connection limit: Why does WCF limit concurrent connections to 5?