Ruby Benchmarking Accuracy - Branch Prediction at its finest?

Question

So this morning I decided to play around with Benchmarking for the first time.

I was curious about the speed different between code with "do-end" block formatting vs. "{ }" formatting.

So I stored the Benchmark code in a Proc so I could call it multiple times consecutively:

n = 100_000_000
bmp = Proc.new do
  Benchmark.bm do |x|
    x.report {n.times {a = "1"}}
    x.report {n.times do; a = "1"; end}
  end
end

My results we're expected when I ran it once.

>> bmp.call
  user     system      total        real
1.840000   0.030000   1.870000 (  1.874507)
1.860000   0.050000   1.910000 (  1.926101)
=> true

But then ran it again.

>> bmp.call
  user     system      total        real
1.870000   0.050000   1.920000 (  1.922810)
1.840000   0.000000   1.840000 (  1.850615)

To me this looks like the exact opposite of what I'm expecting. I am familiar with the concept of Branch Prediction. Is this a classic example of Branch Prediction? If not, what? Is there anyway to prevent any inaccuracies like this (if this is even considered one)?

EDIT: I did run this code over 30 times, after some suggestions. Frequently it would alternate between the two results. The sample of the data is found here:

gist.github.com/TheLarkInn/5599676

Answer 1

First of all, your benchmark is utterly pointless. The difference between the do / end syntax and the { / } syntax is just that: syntax. There is no semantic difference. Ergo, there cannot possibly be any runtime performance difference whatsoever between the two. It's just not logically possible. You don't need to benchmark it.

The only performance difference that could exist, is that one takes longer to parse than the other. However, none of the two is harder to parse than the other. The only difference is precedence. Therefore, there very likely isn't any performance difference in parsing, either.

And even if there were a performance difference in parsing, your benchmark wouldn't show it. You are using a benchmark written in Ruby, but in order to run Ruby code, the Ruby execution engine has to parse it first, which means that parsing will already have happened, before your benchmark even starts. So, even if your benchmark weren't pointless, it would still be useless, since it cannot possibly measure the performance difference in parsing.

As to your question about Branch Prediction: there are no branches in your code, there is nothing to predict.

BTW: even if your benchmark was intended for a different purpose, it still wouldn't be measuring anything, since at least the more advanced Ruby implementations would recognize that your blocks are essentially no-ops and simply optimize them away. And even if they aren't optimized away, all they are measuring is memory allocator performance (allocating a couple hundred megabytes of tiny String objects), not the performance of blocks.

Answer 2

Just a quick primer on stats:

I'm not sure if two runs is enough to spot a trend. What if there was a difference in system load between the two test blocks the second time you ran it?

A rule of thumb for determining a statistical difference between two samples is that 30 or more data points will give you a statistically relevant result.

I'd run your tests at least that many times, store the results for the two versions independently, and then compare them internally to ensure they're consistent, before comparing the two sets to one-another.

It could be that your initial premise is incorrect :)