Signalling NaN was corrupted when returning from x86 function (flds/fstps of x87)

Question

I have strange behaviour with x86 (32-bit) linux gcc. I generate signalling NaN using gcc's builtin __builtin_nansf(""), which generates 0x7fa00000. After returning this value from function as float, it is modified into 0x7fe00000. There is short example:

#include <stdio.h>
float f = __builtin_nansf("");
float y;

float func (void)
{
  return f;
}

int main (void)
{
  printf("%x\n", *((int*)&f));
  y = func();
  printf("%x\n", *((int*)&y));
}

Program compiled with gcc-4.6.2 program.c, its output:

7fa00000
7fe00000

Gdb:

(gdb) p/x f   
$2 = 0x7fa00000
...
(gdb) si
0x08048412 in func ()
1: x/i $pc
0x8048412 <func+14>:    flds   -0x4(%ebp)
(gdb) x/x $ebp-4
0xbfffeb34: 0x7fa00000
(gdb) si
(gdb) info all-regis
st0            nan(0xe000000000000000)  (raw 0x7fffe000000000000000)
... //after return from func
(gdb) si
0x0804843d in main ()
1: x/i $pc
0x804843d <main+38>:    fstps  0x804a024
(gdb) si
(gdb) x/x 0x804a024
0x804a024 <y>:  0x7fe00000

Why my signalling NaN is modified? How can I prevent this modification?

Answer 1

I'm not sure you can prevent this. Loading an sNaN on x87 typically raises an INVALID exception, and then converts the value to a qNaN, by setting the msb of the (23 bit) mantissa. That is, OR'ing with 0x00400000.

From the Intel® 64 and IA-32 Architectures Software Developer Manuals, Vol 1, 4.8.3.4 describes sNan/qNan handling. Chapter 8 deals with X87 FPU programming. Volume 3, 22.18 also describes how NaNs are handled by the X87 FPU.

I don't see any bits in the X87 control word that will yield the behaviour you desire for sNaN propagation.

Answer 2

After doing google search for "gcc 7fa00000" I have found the bug 57484 in GCC's bugzilla http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57484 with several useful comments.

Uroš Bizjak (i386 cpu port maintainer in GCC) says in comments 11,12,14 and in last one, that x86 ABI and x86-32 ABI are not designed to fully support IEEE754 standard on x87 and the "issue is unfortunately unfixable":

The ABI is just wrong for the underlying x87 hardware as far as NaNs are concerned.

This issue is unfortunately unfixable. x87 and x86-32 ABI are just not designed to handle all details of IEEE 754 standard.

According to Uroš, when using legacy x87 on x86 gcc target, loads of float and doubles from memory to the x87 FP registers (stack) is considered as format conversion with changing of signaling NANs (sNAN) into quiet NANs (qNAN). The -msse2 -mfpmath=sse option set will help to do all math evaluations in SSE2, but function still returns FP value via x87 stack:

$ gcc-4.6.3 -msse2 -mfpmath=sse test.c -o sse2math.out
$ objdump -d sse2math.out 

./c.out:     file format elf32-i386
...

08048404 <func>:
 8048404:   55                      push   %ebp
 8048405:   89 e5                   mov    %esp,%ebp
 8048407:   83 ec 04                sub    $0x4,%esp
 804840a:   a1 14 a0 04 08          mov    0x804a014,%eax
 804840f:   89 45 fc                mov    %eax,-0x4(%ebp)
 8048412:   f3 0f 10 45 fc          movss  -0x4(%ebp),%xmm0
 8048417:   f3 0f 11 45 fc          movss  %xmm0,-0x4(%ebp)
 804841c:   d9 45 fc                flds   -0x4(%ebp)
 804841f:   c9                      leave  
 8048420:   c3                      ret    

After adding one more option -mno-fp-ret-in-387 (full set is -msse2 -mfpmath=sse -mno-fp-ret-in-387), x87 fp registers are not more used to pass float return:

08048404 <func>:
 8048404:   55                      push   %ebp
 8048405:   89 e5                   mov    %esp,%ebp
 8048407:   a1 14 a0 04 08          mov    0x804a014,%eax
 804840c:   5d                      pop    %ebp
 804840d:   c3                      ret    

But the -mno-fp-ret-in-387 option will change ABI, and may broke many libraries.