I have detected a strange segmentation fault in my code and I would like to hear your opinion if that could be a GCC bug or is just my fault!
The function looks like that:
void testMMX( ... ) {
unsigned long a = ...;
unsigned char const* b = ...;
unsigned long c = ...;
__asm__ volatile (
"pusha;"
);
__asm__ volatile ( "mov %0, %%eax;" : : "m"( a ) : "%eax" ); // with "r"( a ) it just works fine!
__asm__ volatile ( "add %0, %%eax;" : : "m"( b ) : "%eax" );
__asm__ volatile ( "mov %0, %%esi;" : : "m"( c ) : "%eax", "%esi" );
__asm__ volatile (
"sub %eax, %esi;"
"dec %esi;"
"movd (%esi), %mm0;"
"popa;"
);
}
If I compile this with -O0 it just works fine. But it SegFaults with -O1 and -O2. It took me a long time to figure out that this segfault was caused by frame pointer omission. The pusha instruction increases the stack size by 4*8=32 byte (x86_32) and therefore ESP should be increases as well. But gcc does not recognize this. If I add the ESP fix manually
__asm__("add $32, %esp")
or use the "-fno-omit-frame-pointer" flag in gcc I can compile and run it with -O1 and -O2 without any errors!
So my question now is: why does gcc not adjust the ESP with any push/pop inline assembler operations if frame-pointer-omission is enabled? Is this a gcc bug? Is gcc even capable of detecting this? Am I missing something?
It would be very interesting to solve this.
Thanks in advance!