Can I rely on absolute cost of valgrind/callgrind to measure and compare different implementations?

Question

Now I am using valgrind/callgrind to meaure and compare different algorithm implementations. Suppose there are two implementations for an algorithm, and the processing chains are as follows:

void main()
{
    //Procedure 1
    fun1();
    //Procedure 2
    fun2();
    //Procedure 3
    Algorithm_imp_1();  //Algorithm_imp_2(); 
   //Procedure 4
    fun4();
}

The difference between these two implementations lies in the third procedure, where two different algorithm implementations are performed. In order to see which implementation is better, I am now using valgrind/callgrind with the help of kachegrind.

As far as I can image, if Algorithm_imp_1() is more efficient than Algorithm_imp_2(), its two indicators: one is the absolute time that have been used for running the program , and the other is the percentage of the time the third procedure takes, should be smaller. However, what I have obtained with valgrind are very confusing:

Method 1 whole procedure: overall circle 106858896849   
        Algorithm_imp_1():  circle 2971828457(3.03%)
Method 2 whole procedure: overall circle 137889090577
        Algorithm_imp_2(): circle 351826053(0.26%)

Since the whole procedure for the both methods are the same expect the third part, if the consumed time percentage of the third part is small, we can expect the overall time for running the program should also be small. However, what we observed above are contradictory. I was wondering what's wrong with my analysis. Thanks!

Answer 1

The shell has a built-in time command. If you use that, it will display wall clock as well as user and system times.

linux> /bin/sh
$ time a.out
    0m1.01s real     0m0.00s user     0m0.00s system

If you need a better resolution than 0.01 second, you can use getrusage and gettimeofday.

When profiling a function, it is best to place instrumentation as near the function as possible. Using something like valgrind will most likely mask what is going on within the function.

The signal to noise ratio of the measurement has to be high for it to be useful.

linux> cat a.c
#include <stdio.h>
#include <unistd.h>
#include <sys/time.h>
#include <sys/resource.h>

/*
 * Time elapsed in milli-seconds. Tweak this to get micro-seconds. 
 */

int
time_diff(struct timeval *t2, struct timeval *t1)
{
    int r;

    r = (t2->tv_sec - t1->tv_sec)*1000 + t2->tv_usec/1000 - t1->tv_usec/1000;

    return(r);
}

int
main()
{
    struct rusage r1, r2;
    struct timeval t1, t2;

    (void)gettimeofday(&t1, 0);
    (void)getrusage(RUSAGE_SELF, &r1);

    sleep(1); /* Function to be profiled. */

    (void)getrusage(RUSAGE_SELF, &r2);
    (void)gettimeofday(&t2, 0);

    printf("real = %d (ms), user = %d (ms), system = %d (ms)\n", 
            time_diff(&t2, &t1),
            time_diff(&r2.ru_utime, &r1.ru_utime),
            time_diff(&r2.ru_stime, &r1.ru_stime));

    return(0);
}

When you run the above, you should see something similar to below.

linux> a.out
real = 1000 (ms), user = 0 (ms), system = 0 (ms)