QTAGENT.EXE Using All Available CPU

Question

I'm pretty new to load testing with VS, but I think I've set everything up normally, and I'm having a problem where execution of my Load Test quickly spikes to 100% CPU utilization. It seems likely that this will affect my test results and I assume it's not the anticipated behavior.

My hardware platform is a beefy developer machine: a few months old with a quad-core, 12GB RAM, 2 IDE drives and a fast NIC. I can elaborate if necessary, but usually I'd say it screams.

I'm testing an MVC2/3 application that runs locally on Win7 x64, against MSSQL 2008, using VS2010 SP1, C#/.NET 4.

The underlying test is a Unit Test that exercises a single MVC controller method, which uses Unity and other Ent-Lib libraries to pull a collection of clients back from the database, according to the user's rights. Typical execution time for the Unit Test is 0.4 to 0.6 seconds.

The Load Test is set to expose the performance problem and runs for 10 minutes:

• Initial User Count: 1
• Step Duration: 30
• Step User Count: 1

Observations:

1. For the first 30 seconds as a single user, execution is a bit erratic. The first test takes 3 sec, which is normal warm-up. The next 15 or so take about .4 sec, which is perfect. Just after 10 seconds, though, tests start running for 5 sec. This is before the 2nd virtual user comes on. As additional virtual users come on, test times increase gradually, which is to be expected, though the runtimes still seem long.

2. More concerning is that CPU utilization spikes around user 6 (it was averaging around 20% prior across all 8 cores). At user 6 and 7, CPU goes to 70% and is pegged at 100% on 8 cores by user 8, and it stays there until the test ends after 10 minutes. Needless to say, for a normal stepped test, I'd like to start with 10 concurrent users and go up to 100 or more, but my CPU starts at 100%, and the results are surely inaccurate.

The cuprit is clearly QTAGENT.EXE, as I can see from the executing tasks. It uses every spare CPU cycle from what I can see. I've run this Load Test in 32-bit mode as well, and QTAGENT32.EXE runs similarly.

I'm at a loss for why I'm experiencing this problem, let alone what to do about it. Any proposals or solutions are appreciated. TIA!

EDIT

I found that a workaround to my problem is to start the load test by clicking "Debug Test" instead of "Run Test." This doesn't really make any sense to me, in fact it's like the exact opposite of what I'd expect; however, I can't deny it's consistently running my tests with 100s of users and only semi-thrashing my CPU (~50% average with spikes).

Answer 1

So it came down to user error. Turns out the controller I was testing was ultra-resource-intensive. I read about this as the likeliest cause of high CPU usage elsewhere, but I assumed (oops) that 8 concurrent users hitting this particular controller couldn't possibly peg my CPU. No, in fact, our code is extremely expensive to run, and 8 users requesting the page at the same time will max it out. Thanks for all the suggestions.

Answer 2

QTAgent is the code that does all the load test work. It loops through each virtual user executing the tests.

One reason the test agent CPU gets pegged is that your website is firing back results really fast. A good way to simulate this is to load test a static html page, or requesting a page that is well cached by IIS.

Check the requests per second value for the test. That value is a reasonable measure of how much work the QTAgent is required to do.

For each test, your QTAgent has to process and send an html request, record the received test and store the results, along with all the performance monitoring stats.

I would recommend as a next step to add a 1 second delay between each unit test, that will hopefully let you load on a few more virtual users.

That said, running load tests on a web app hosted on the same machine cannot handle very many virtual users before the CPU pegs out.

Can you post the requests per second stats?

Answer 3

Maybe something here might help: https://serverfault.com/questions/16005/what-affects-sql-connection-speed

Other ideas:

• Is SQL Server on the local box? Running the test on the same box might cause problems since SQL Server and QTAgent will be competing for processing time. Each test thread will be matched by one database thread, and unless your unit tests are calling the database asynchronously, they will be busy-waiting while the database responds. At 8 users, that's 8 unit test threads and 8 database threads... maybe 16 threads going full speed is enough to slam the CPU.

• Does your test close the connection after each iteration, or are you possibly leaving connections open? You could check the SQL Server connection count in Perfmon to determine that.

• Is it SQL Server Express? Maybe it's not as robust as the other versions.