When should libm functions set errno?

["Joseph S. Myers" <joseph at codesourcery dot com>]

This is a followup to 
<https://sourceware.org/ml/libc-alpha/2013-05/msg00132.html> where I 
described goals for libm function results, exceptions and errno, as well 
as to various open glibc bug reports relating to missing errno settings 
from libm functions.  Other previous relevant messages include 
<https://sourceware.org/ml/libc-alpha/2013-07/msg00444.html>, where I 
discuss possible function variants glibc could provide for different 
requirements, and 
<https://sourceware.org/ml/libc-alpha/2013-04/msg00767.html>, discussing 
other error handling variants (_LIB_VERSION and matherr).  It's 
independent of the questions of whether (a) to deprecate _LIB_VERSION and 
matherr so new programs cannot use them (as far as that's possible) and 
(b) to eliminate _LIB_VERSION runtime checks from the code paths for new 
programs.

ISO C90 specified errno setting for domain errors and overflows and made 
it implementation-defined whether errno is set to ERANGE for underflows.  
C99 introduced a quiet change that could break C90 applications, by making 
errno setting optional, with math_errhandling specifying whether the 
implementation indicates errors by errno, exceptions or both.  C11 split 
the concept of pole errors out from overflows (they set different 
exceptions, but the same errno value).

Where math_errhandling & MATH_ERRNO is nonzero (as for glibc unless 
-ffast-math; there isn't a predefined macro for -fno-math-errno but there 
probably should be), C99/C11 require errno setting for: domain errors; 
overflows in default rounding mode; pole errors.  errno setting for 
underflow is implementation-defined, as in C90.  Nothing is stated 
regarding errno setting for overflows in nondefault rounding modes.  It is 
explicitly stated that <complex.h> functions may set errno but are not 
required to.

glibc's documentation states:

  When a math function suffers a domain error, it raises the invalid
  exception and returns NaN.  It also sets @var{errno} to @code{EDOM};
  this is for compatibility with old systems that do not support @w{IEEE
  754} exception handling.  Likewise, when overflow occurs, math
  functions raise the overflow exception and return @math{@infinity{}} or
  @math{-@infinity{}} as appropriate.  They also set @var{errno} to
  @code{ERANGE}.  When underflow occurs, the underflow exception is
  raised, and zero (appropriately signed) is returned.  @var{errno} may be
  set to @code{ERANGE}, but this is not guaranteed.

The Linux man-pages collection has a math_error.7 page with its own 
description of error handling, which is out of date (claiming 
math_errhandling is not supported), has a typo where it says 'an 
"overflow" (FE_UNDERFLOW) floating-point exception' and may have other 
issues (not checked).

A typical way for a libm function to check for whether errno setting is 
needed is to check for the result not being finite and use special-case 
code if so.  (A typical bug in this code is that it's written as if 0 is 
not finite, so the cases that try to set errno for underflow don't 
actually get called because of the check of __finite without checking for 
0 as well.)

Note that this approach does not set errno for cases of underflow where 
the result is subnormal (or the least normal), or overflow in directed 
rounding modes where the result is the largest normal rather than 
infinite.  Note also that ISO C does not require errno settings in those 
cases even when math_errhandling & MATH_ERRNO is nonzero, and that they 
are outside the scope of C90 compatibility (C90 did not have alternative 
rounding modes, and did not consider subnormals at all).

My proposal is that we should officially bless this implementation 
approach (fixed to properly consider 0 as underflowing).  That is, we 
should document that errno is only expected to be set on overflow where 
that overflow is to infinity, document in which cases we intend errno to 
be set on underflow while keeping the caveat about it sometimes being 
missing, and consider the goal in fixing bugs about missing errno on 
underflow to be only to fix it for underflow to 0.  Comments?

(This would affect the validity of some of the reported bugs, e.g. 6785, 
6786, 6795, though in various cases it's possible the bug is actually 
valid with a different testcase for underflow.)

-- 
Joseph S. Myers
joseph@codesourcery.com