Visual Studio Load Test to simulate many users in Data Driven fashion?

Question

I'm trying to load test a web service.

I've got a simple method that takes 2 params that get sent to the web service and returns the response, and I created a Unit Test from this method.

Because I want need to test various different inputs to effectively test the service, I setup a CSV with ~1000 rows as a DataSource for the Unit Test.

I can run this one Unit Test and it will spin through all the rows, sequentially calling the web service with different values.

However, this is not what I need. I need each row to correspond to a different user on a different thread, using the Load Test configuration to include think times, ramp up users with step load, test mix config, etc.

I can achieve this by removing my data source from the unit test and hard-coding values .. but this is a fundamentally flawed test: In order to really test the web service, I need each user to be sending different values and getting different results.

...

So, how can I hook a data source to a load test and have that load test start each user's instance of a unit test with different values?

Answer 1

I ended up using this answer as a guide: https://stackoverflow.com/a/7813465/237723

Rather than create a Unit Test from the console app method and Load Test that Unit Test, I created a simple ASP.NET web form that took the inputs and called the web service.

I recorded a Web Performance Test using this new form, and created a Load Test to run that WPTest.

---

1. After recording the test, I added the CSV as a DataSource to this test.

2. There were 2 'Requests' recorded: the initial GET & subsequent POST. Make sure you leave them both! I removed the favicon.ico request because it didn't exist. (these precautions may be avoidable)

3. By expanding the POST Request, I modified the properties of the TextBox parameters that corresponded to my 2 web service inputs to get their values from the appropriate column in the CSV.

4. I changed the DataSource Access method to 'Do Not Move Cursor Automatically' (You have to expand the DataSource to the table and edit its properties via right-click/F4.)

5. I then created the WebTestPlugin (from the linked answer) that manually moves the cursor according to the UserID (an int) being run. This will correspond to the user instance the Load Test spins up according to the step plan. After you create this class, build the project and then add it to your Web Performance Test.

---

public class webtestplugin : WebTestPlugin
{
    public override void PreWebTest(object sender, PreWebTestEventArgs e)
    {
        base.PreWebTest(sender, e);
        e.WebTest.MoveDataTableCursor("DataSource1", "addresses#csv", e.WebTest.Context.WebTestUserId);                        
    }
}

Answer 2

Just to answer your question, i'm pretty sure this can't be done out of the box of MSTest and NUnit (this approach won't stick in this scenario).

---

But, IMHO, just don't go there... From my experience, simulating ~1000 users out of a single machine will produce bad results, because the test will encounter all sorts of client limits - thread pool issues, outgoing and incoming traffic issues, etc. I'm not saying this can't be overcome, but it's twisted enough to consider a different approach.

I actually don't recommend using load test tools (there are many of them out there) in this case, as it is simple enough to write a little tool of your own and skip the configuration problems and learning curves of a 3rd party.

What I do recommend, is writing a tool of your own, and running it from seperate machines. It doesn't have to be run by a testing framework (I can't get myself to call it a unit test, because it's not), a console app will do the trick. Here's some code to get you started:

private ConcurrentBag<string> logs = new ConcurrentBag<string>();

public void GetLoad(int numberOfUsers, List<string> myParams)
{
    var users = new string[numberOfUsers];
    for (int i = 0; i < numberOfUsers; i++)
    {
        users[i] = string.Format("LoadTest{0}", i + 1);
    }

    var userThreads = new List<Thread>();
    for (int i = 0; i < numberOfUsers; i++)
    {
        int index = i;
        userThreads.Add(new Thread(()=> CallWebService(users[index], myParams[index])));
    }

    Parallel.ForEach(userThreads, thread=>thread.Start());
    foreach (var userThread in userThreads)
    {
        userThread.Join();
    }
    var outputFilename = string.Format("LoadTest.{0}Users.txt", numberOfUsers);
    File.AppendAllLines(outputFilename, logs);
}

Answer 3

The image shows show where to make the modification.