How to measure the time taken by executing a piece of code in a reliable way?

Question

The question of how to measure execution time has been asked plenty of times on SO, see here or here for example. However these questions mostly focus on which timing functions to use.

What I am interested in is how to make the results reproducible. For example, due to multitasking the execution of a tested code can be interrupted by some background process affecting the results. To overcome this I've seen some benchmarks do multiple runs and take the best time (in addition to running the code multiple times in a loop).

Are there any other suggestions or ideas on how to make the results more reliable and reproducible?

Answer 1

First you may want to separate the code that you want to speed test from calls to the code that you do not want to test. Especially make sure such code will not do file IO, display output, logging, console output etc. (Of course, except if you test the IO code itself)

You can also load all data needed to do test run in advance so data loading is not measured.

The good trick is to test as little as possible. If you can track down to those few lines that make most difference, then you can find a way to test them separately from the rest of code. Maybe even copy/paste them into a new function carefuly designed for this sole purpose.

If you are really set to do hi-definition time counting you can also try to use OS provided function to enter real-time or high priority CPU mode.

In some cases you might want to see the generated low level machine code, and then use the reference data from CPU manufacturer (or VM implementation) to calculate how many 'cpu-ticks' the code in question will take. Since many cpu/vm instruction have a variable execution time depending on pipeline/operands/cpu model/etc, even this approach will provide approximate value.

So the best advice is to correctly set the error margin for your measurements. Normally I personally consider this 10% when I start to time-test some code. Then this value can (usually) decrease. But it is never 0%. So when your program outputs "Execution Time: %i ns" you can (and probably should) always print "+- X ns".

Answer 2

Please check the code it may be useful in Java

package com.test.stackoverflow;

public class Sample {

    /**
     * @param args
     */
    public static void main(String[] args) {
        // TODO Auto-generated method stub
        long startTime = System.currentTimeMillis();


        for(int i=0;i<10000; ++i)
        {
            System.out.println("You ae executng Looop");
        }
        long endTime = System.currentTimeMillis();


        long diffTime = endTime - startTime;

        System.out.println("Start Execution at in Time (Seconds ) "+ startTime * 0.001);

        System.out.println("End Execution at in Time (Seconds ) "+ endTime  * 0.001);

        System.out.println("Difference in Time (Seconds ) "+ diffTime * 0.001);

    }

}