How do I Benchmark RESTful Service with Variable Parameters?

Question

I'm currently working on benchmarking a RESTful service I've made, and part of that is making sure it runs in a reasonable amount of times for a large array of parameters. For example, let's say I have RESTful API of the form some_site.com/item?item_id=y. In that case to be sure my service is working as fast as I'd like it to work, I'd want to try out many values for y one by one, preferably coming from some text file. I can't figure out any way of doing this in ab or httperf. I'm open to using a different benchmarking program if I have, but would prefer something simple and light. What I want to do seems like something pretty standard, so I'm guessing there must already be a program that let's me do it, but an hour or so of googling hasn't gotten me an answer. Ideas?

Answer 1

Answer: Jmeter (which is apparently awesome). This faq explains how to do it. Hopefully this helps someone else, as it took me like a day of searching to figure this out.

Answer 2

I have just had some good experience with using JavaScript (via BSF/Rhino) in JMeter.

I have put one thread group in my test plan and stick a 'Simple Controller' with two elements under it - 'HTTP Request' sampler and 'BSF PreProcessor'.

Set BSF language to 'javascript' and either type the code into the text box or point it to a file (use full path or relative to CWD of JMeter process).

/* Since `Math.random()` gives us float, we use `java.util.Random()`
 * see: http://docs.oracle.com/javase/7/docs/api/java/util/Random.html */
var Random = new Packages.java.util.Random();

var min = 10-1;
var max = 2;
var maxLines = (min)+Random.nextInt(max-min);

var s = '';

for (var d = 0; d <= maxLines; d++) {
  s += d.toString()+','+Random.nextInt(1000).toString()+'\n';
}

// s => '0,312\n1,104\n2,608\n'

vars.put('PAYLOAD', s);

Now I can refer to ${PAYLOAD} in the HTTP request!

You can generate JSON, but you will need to upgrade jakarta-jmeter-2.5.1/lib/js-1.6R5.jar with the newest version of Rhino to get JSON.stringify and JSON.parse. That worked perfectly for me also, though I thought I'd put a simple example here.

You can use BSF pre-processor for URL params as well, just set another variable with vars.put('X', 'some value') and pass it as ${X} in the request parameter.

This blog post helped quite a bit, by the way.