Unexpectedly large number of TLB misses in simple PAPI profiling on x86

Question

I am using the PAPI high level API to check TLB misses in a simple program looping through an array, but seeing larger numbers than expected.

In other simple test cases, the results seem quite reasonable, which leads me to think the results are real and the extra misses are due to a hardware prefetch or similar.

Can anyone explain the numbers or point me to some error in my use of PAPI?

int events[] = {PAPI_TLB_TL};
long long values[1];
char * databuf = (char *) malloc(4096 * 32);

if (PAPI_start_counters(events, 1) != PAPI_OK) exit(-1);
if (PAPI_read_counters(values, 1) != PAPI_OK) exit(-1); //Zeros the counters

for(int i=0; i < 32; ++i){
    databuf[4096 * i] = 'a';
}

if (PAPI_read_counters(values, 1) != PAPI_OK) exit(-1); //Extracts the counters

printf("%llu\n", values[0]);

I expected the printed number to be in the region of 32, or at least some multiple, but consistently get a result of 93 or above (not consistently above 96 i.e. not simply 3 misses for every iteration). I am running pinned to a core with nothing else on it (apart from timer interrupts).

I am on Nehalem and not using huge pages, so there are 64 entries in DTLB (512 in L2).

Answer 1

Based on the comments:

• ~90 misses if malloc() is used.
• 32 misses if calloc() is used or if the array is iterated through before hand.

The reason is due to lazy allocation. The OS isn't actually giving you the memory until you touch it.

When you first touch the page, it will result in a page-fault. The OS will trap this page-fault and properly allocate it on the fly (which involves zeroing among other things). This is the overhead that is resulting in all those extra TLB misses.

But if you use calloc() or you touch all the memory ahead of time, you move this overhead to before you start the counters. Hence the smaller result.

As for the 32 remaining misses... I have no idea.
(Or as mentioned in the comments, it's probably the PAPI interference.)

Answer 2

The reason for this is possibly because you are jumping by 4096 in every loop iteration:

for(int i=0; i < 32; ++i){
    databuf[4096 * i] = 'a';
}

There is a good chance that you get a cache miss for every access.