Benchmarking functions multiple times - All calls after first are instruction cached?
-
26-10-2019 - |
Question
I'm writing a small benchmarking library in C which is used to benchmark single functions. The way it works is that you supply the benchmarking function with a pointer to a void
function without parameters and the number iterations. The function then returns a struct
with information about
However when looking at the individual results I see that the first call takes up a lot of time and then the calls after takes up very little time.
So I'm wondering:
- Is this due to instruction caching?
- If yes to first question, how does benchmarking tools generally conform to this? Is the first call excluded?
- If yes to first question, is there a case when caching is not applied to a function?
- If yes to first question, does caching occur on whole functions or segments of a function?
- If yes to first question, is there anything else I should consider that I should read up on and understand better?
Data and code
The function which retrieves the time:
double currentTime()
{
struct timeval time;
struct timezone timezone;
gettimeofday(&time, &timezone);
return time.tv_sec + time.tv_usec * 1e-6;
}
Test result:
0.000319
0.000000
0.000000
0.000000
0.000001
0.000000
0.000000
0.000000
0.000000
0.000000
Total time: 0.000320
Average time: 0.000032
Worst time: 0.000319
Best time: 0.000000
Result without optimizing flag:
13.425430
13.349757
13.482863
13.129472
13.020705
13.672982
13.027595
13.139602
13.028962
13.107892
Total time: 132.385260
Average time: 13.238526
Worst time: 13.672982
Best time: 13.020705
The function being tested which produces this result:
void test()
{
unsigned int i = 0;
while(i++ < UINT_MAX){}
}
Make file:
CC = gcc
MAIN = main.c
SOURCES = lib/tb_time_handling.c lib/tb_rendering.c tb_benching.c
OUTPUT = main
FLAGS = -Wall -pedantic -O2
all: main
main:
$(CC) $(MAIN) $(SOURCES) -o $(OUTPUT) $(FLAGS)
Github repo with all code:
https://github.com/Ancide/TinyBench
Edit: Forgot to mention compiler and compiler flags
Edit 2: Added git repo with all code in case someone wants to see everything
Edit 3: Added results withouth O2 flag
Solution
This is normal in virtual memory operating systems. The first call to a function tends to generate a page fault which loads the code from the executable file into RAM. If you are interested in actual code perf then you ignore the first call. If you are interested in realistic perf measurements then you don't ignore it.