Re: [PATCH] Fix setting function return values for x86 targets

[Mark Kettenis <kettenis at wins dot uva dot nl>]

   Date: Tue, 02 Jan 2001 11:40:14 -0800
   From: Michael Snyder <msnyder@redhat.com>

   Mark Kettenis wrote:
   > 
   > FYI, this patch fixes some problems uncovered by Michaels new
   > gdb.base/return2.exp tests for the x86.  STORE_RETURN_VALUE didn't
   > handle long long's and floating point values correctly.  There are
   > still some problems with floating point values (similar to the
   > problems with extracting return values, such that some of the new
   > tests still fail.  I'll try to come up with a fix, which should be
   > possible for native GDB (or x86 x x86 cross).
   > 
   > Tested on i586-pc-linux-gnu & checked in.

   Looks good.  Thanks Mark.

Except that the "fix" I had in mind to deal with returning floating
point values doesn't really work.  In fact, the way the x86 FPU
handles floating point numbers, means that on every x86 target that
returns floating point numbers in %st(0) (AFAIK that's all of them)
you'll see the following failures:

   FAIL: gdb.base/return2.exp: float value returned successfully
   FAIL: gdb.base/return2.exp: double value returned successfully

The problem is that gdb.base/return2.exp tries to return a floating
point number with all bits set to one, which is one of the many
possible encodings for a QNaN.  When storing a floating point return
value on the FPU stack, it will be converted into extended precision,
and when the caller reads the return value from the stack it is
converted back into a single or double precision number.  In this
process a NaN will be "normalized", i.e. the many possible
representations of a QNaN will be mapped onto one single
representation.  As a consequence 

   float_resultval != testval.float_testval

and the test fails.

I'm inclined to turn those two in XFAILs for all x86 targets.  Would
that be acceptable?  The alternative would be to change the test to
return a normal floating point number instead of a NaN.

Mark