.NET Core: Does sending the same request multiple times as part of a stress test/load test make the test unreliable?

Question

I've got a .NET Core application that takes HTTP requests. I'm a little bored, so I decided to stress-test it using West Wind Web Surge, and putting an example request as the body.

My question is, since the request is exactly the same every time, am I seeing artificially high performance because the application has already processed a similar request? Is it likely something is saved in-memory rather than recalculated every time, and I'm not aware of it? Is it worth the effort to build an application that will generate requests randomly to get a truer picture? My code is pretty standard, nothing fancy going on.

I know it's a generic question, but I don't really know where to look to find the answer.

Answer 1

Yes. (sort of)

You can have all kinds of caching between your client and the eventual response. If you are running your tests against an environment and network that you don't fully control, such as a live deployment where various networking and deployment teams may be involved, then its likely that there will be some caching policies in place.

Now, do you want to test the performance with or without caching? if you test it without then are your results useful? After all the live system will be using a cache.

If you test it with, then again, does the amount of caching match what you would see if live? You presumably don't expect 100% identical requests in live.

What you need to do is mimic the live system as best you can. This means generating pseudo or replaying live traffic against a system which is deployed to the same sort of hardware as your live system.

Answer 2

Well-behaved load test is not hammering single URL, this form of test doesn't make a lot of sense (unless you're load testing .NET Core or IIS themselves), it represents real life user sitting behind real browser with all related stuff like:

• Headers
• Cookies
• Behavior with regards to embedded resources (images, scripts, styles, fonts, etc.) - real browsers download them using 6 parallel threads and respecting Cache-Control headers
• Cache itself, real browsers do download embedded resources, but do this only once, on subsequent requests the resources are being returned from cache and no actual request is being made
• AJAX requests - although none of load testing tools can actually execute client-side JavaScript if your application uses AJAX technology and it generates network requests - you need to simulate this as well as browsers execute JavaScript in asynchronous manner and the majority of load testing tools cannot kick off extra threads

And last but not least, if your application assumes different use cases by different user roles you should consider creating different independent groups of users which will map to your application use cases (i.e. anonymous users just crawling, logged in users filling some forms, etc) and the workload will more or less represent real-life distribution.

So the main idea of good load test is that this test should be realistic, otherwise it does not make a lot of sense as you will not be able to tell for sure how many simultaneous users you application supports providing reasonably low response time, when it breaks, does it recover when load decreases, etc.