Perfomance Test reporting with accurate TPS

Question

I need to complete some performance tests on SOA appliances with side cache. I have developed a simple application to generate SOAP/HTTP traffic but now i need some way of monitoring the E2E applications performance.

One vital metric i require is an accurate figure for the Transactions Per Second, as well as e2e response time.

I've used soapUI and Loadui but just do not believe the reported TPS figures as they seem very high, e.g. > 1300 TPS.

can anyone recommend a method to measure TPS that is "fool proof"?

Answer 1

I'd suggest cross checking SoapUI's numbers against the logs from your server (count the number of lines with the same second), or cross check this way:

Time the test run yourself.
Verify that the number of transactions SoapUI cites are accurate (logs or another measure on the server itself). Divide trans count by seconds.

In the past I've done this and found SoapUI to be pretty reliable.

One thing to keep in mind in terms of whether your numbers are as good as they can be is whether or not you might need to simultaneously run soapui from more than one machine. I suggest monitoring the CPU, memory, bandwidth, etc on the SoapUI machine. If any of these get rather high, run the test on two machines simultaneously with very close to the same start and stop times and then you can safely add the two TPS numbers.