Could high levels of .NET Contention Rate and CLR Security RT Checks be caused by excessive SQL calls

Question

I am getting a report from our QA team that a particular part of an ASP.NET app is showing very high CPU usage with just 10 simultaneous users. The other symptoms that are being reported are:

• High rate of Contention Rate/sec (.NET CLR LocksandThreads counter).
• Observed the application spending a 29% of time performing runtime checks (% time in RT checks - .NET CLR Security)

Looking through the code and running SQL Trace while rerunning the individual AJAX calls that the screen makes I am seeing that some of the calls make lots of separate synchronous SQL calls (60+ in one case), some very fast and some fairly slow.

Additional detail: this code uses Windows Authentication.

So I am clear that the high number of separate SQL round trips is probably a problem from a performance standpoint, but could the fact that the code is having to wait for a response from the SQL box over and over before it can complete be responsible for a high contention rate and % time in RT checks? Are there additional questions I should be asking?

Answer 1

If your application is installed on a separate server than the DB, the symptoms you are describing states that the CURRENT bottleneck has nothing to do with the DB.

The reason I've highlighted "Current" bottleneck is because performance improvement is an iterative process, you find a bottleneck, fix it and hit the next bottleneck (which could have a whole different symptoms), and so forth until the customer is satisfied.

If you set the CLR Locks and Threads performance counter to monitor only your application process, the results you described would mean that you have a lot of contentions in your application code, and not the DB. Long queries has an opposite symptoms, a lot of IO and waiting threads and CPU is low (as it mostly waits for data to come back from the DB/Network) - while the throughput is still low.

You haven't mentioned what is the contention rate per second, but given that you stated that there are only 10 users connected to the app, than any number will be a big number, and adding 10 more users and the application will render unresponsive.

The next steps will be to open a resource contention profiler (Visual Studio has this functionality built-in), and see which are the lock that causing most of the contentions. On a server side application, locking is generally bad for throughput and should be avoided by all means (use immutable data models, immutable caches, queues, CompareExchange operations, Thread local data - sometimes even at the cost of cloning data, etc...).

Go through your code and see which locks you can get rid off, and then you'll get to the next bottleneck (DB probably :) )

Hope this helps, Ofir.