This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Remove unused libm-test expected-failure mechanism
- From: "Joseph S. Myers" <joseph at codesourcery dot com>
- To: <libc-alpha at sourceware dot org>
- Date: Sat, 4 May 2013 13:43:58 +0000
- Subject: Remove unused libm-test expected-failure mechanism
libm-test.inc has a mechanism for libm-test-ulps files to specify
expected failures as well as ulps values. No architecture uses it, it
can only handle certain kinds of errors (e.g. one of result and
expected value being infinity/NaN and the other not; if the error is
simply different nonzero values, an ulps value could just be specified
directly), in particular not those relating to exceptions or errno
values, and there is no convenient system for generating such expected
failures if (re)generating a libm-test-ulps file from scratch.
We now have a de facto convention of skipping tests known to fail via
#if conditions, or relaxing what's allowed in the case of exception
tests, with a comment referring to the bug filed in Bugzilla; while
libm-test-ulps files can also contain comments, any regeneration of
them at all (e.g. "make regen-ulps"), not just from scratch, will lose
any manually-added comments, so indicating a further advantage of
skipping tests this way rather than using the libm-test-ulps
expected-failure mechanism - the ability to have comments referring to
the associated bug, which don't automatically disappear.
Thus, I propose this patch to remove the unused expected-failure
mechanism. This of course makes the generated libm-test.c smaller and
presumably faster to compile (one fewer function argument in thousands
of generated function calls), as well as simplifying the conversion to
data-driven tests (one fewer bit of information for each test to put
in the arrays of test inputs and then pass to appropriate functions).
Tested x86_64.
2013-05-04 Joseph Myers <joseph@codesourcery.com>
* math/libm-test.inc (noXFails): Remove variable.
(noXPasses): Likewise.
(BUILD_COMPLEX_INT): Remove macro.
(print_screen): Remove xfail argument.
(print_screen_max_error): Likewise.
(update_stats): Likewise.
(print_max_error): Likewise. Update calls to other affected
functions.
(print_complex_max_error): Likewise.
(test_single_exception): Update calls to print_screen.
(test_single_errno): Likewise.
(check_float_internal): Remove xfail argument. Update calls to
other affected functions.
(check_float): Likewise.
(check_complex): Likewise.
(check_int): Likewise.
(check_long): Likewise.
(check_bool): Likewise.
(check_longlong): Likewise.
(main): Don't print noXFails and noXPasses.
* math/gen-libm-test.pl (top level): Don't mention expected
failure handling in comment.
(new_test): Don't handle expected failures.
(parse_args): Don't mention expected failure handling in comment.
(generate_testfile): Don't handle expected failures.
(parse_ulps): Likewise.
(print_ulps_file): Likewise.
(get_failure): Remove function.
(output_test): Don't handle expected failures.
* make/README.libm-test: Don't mention expected failure handling.
diff --git a/math/README.libm-test b/math/README.libm-test
index 7003e5d..cea0392 100644
--- a/math/README.libm-test
+++ b/math/README.libm-test
@@ -78,12 +78,7 @@ Function "yn":
idouble: 6.0000
The keywords are float, ifloat, double, idouble, ldouble and ildouble
-(the prefix i stands for inline). You can also specify known
-failures, e.g.:
-
-Test "cos (pi/2) == 0":
-float: 1
-float: fail
+(the prefix i stands for inline).
Adding tests to libm-test.inc
=============================
diff --git a/math/gen-libm-test.pl b/math/gen-libm-test.pl
index f6ed1a8..a743797 100755
--- a/math/gen-libm-test.pl
+++ b/math/gen-libm-test.pl
@@ -25,7 +25,6 @@
# is a maximal error of a function or a single test.
# $results{$test}{"type"} is the result type, e.g. normal or complex.
# $results{$test}{"has_ulps"} is set if deltas exist.
-# $results{$test}{"has_fails"} is set if exptected failures exist.
# In the following description $type and $float are:
# - $type is either "normal", "real" (for the real part of a complex number)
# or "imag" (for the imaginary part # of a complex number).
@@ -33,8 +32,6 @@
# It represents the underlying floating point type (float, double or long
# double) and if inline functions (the leading i stands for inline)
# are used.
-# $results{$test}{$type}{"fail"}{$float} is defined and has a 1 if
-# the test is expected to fail
# $results{$test}{$type}{"ulp"}{$float} is defined and has a delta as value
@@ -164,22 +161,17 @@ sub get_variable {
}
# Add a new test to internal data structures and fill in the
-# ulps, failures and exception information for the C line.
+# ulps and exception information for the C line.
sub new_test {
my ($test, $exception) = @_;
my $rest;
- # Add ulp, xfail
+ # Add ulp.
if (exists $results{$test}{'has_ulps'}) {
$rest = ", DELTA$count";
} else {
$rest = ', 0';
}
- if (exists $results{$test}{'has_fails'}) {
- $rest .= ", FAIL$count";
- } else {
- $rest .= ', 0';
- }
if (defined $exception) {
$rest .= ", $exception";
} else {
@@ -369,7 +361,7 @@ sub parse_args {
++$current_arg;
}
}
- # Add ulp, xfail
+ # Add ulp.
$cline .= &new_test ($str, ($current_arg <= $#args) ? $args[$current_arg] : undef);
# special treatment for some functions
@@ -457,11 +449,6 @@ sub generate_testfile {
} else {
$line .= '0';
}
- if (exists $results{$fct}{'has_fails'}) {
- $line .= ", FAIL$fct";
- } else {
- $line .= ', 0';
- }
$line .= ");\n";
print OUTPUT $line;
push @functions, $fct;
@@ -522,10 +509,7 @@ sub parse_ulps {
if (/^i?(float|double|ldouble):/) {
($float, $eps) = split /\s*:\s*/,$_,2;
- if ($eps eq 'fail') {
- $results{$test}{$type}{'fail'}{$float} = 1;
- $results{$test}{'has_fails'} = 1;
- } elsif ($eps eq "0") {
+ if ($eps eq "0") {
# ignore
next;
} else {
@@ -591,9 +575,6 @@ sub print_ulps_file {
&clean_up_number ($results{$test}{$type}{'ulp'}{$float}),
"\n";
}
- if (exists $results{$test}{$type}{'fail'}{$float}) {
- print NEWULP "$float: fail\n";
- }
}
}
}
@@ -617,9 +598,6 @@ sub print_ulps_file {
&clean_up_number ($results{$fct}{$type}{'ulp'}{$float}),
"\n";
}
- if (exists $results{$fct}{$type}{'fail'}{$float}) {
- print NEWULP "$float: fail\n";
- }
}
print NEWULP "\n";
}
@@ -652,37 +630,13 @@ sub get_ulps {
? $results{$test}{'normal'}{'ulp'}{$float} : "0");
}
-sub get_failure {
- my ($test, $type, $float) = @_;
- if ($type eq 'complex') {
- # return x,y
- my ($res);
- # Return 0 instead of BUILD_COMPLEX_INT (0,0)
- if (!exists $results{$test}{'real'}{'ulp'}{$float} &&
- !exists $results{$test}{'imag'}{'ulp'}{$float}) {
- return "0";
- }
- $res = 'BUILD_COMPLEX_INT (';
- $res .= (exists $results{$test}{'real'}{'fail'}{$float}
- ? $results{$test}{'real'}{'fail'}{$float} : "0");
- $res .= ', ';
- $res .= (exists $results{$test}{'imag'}{'fail'}{$float}
- ? $results{$test}{'imag'}{'fail'}{$float} : "0");
- $res .= ')';
- return $res;
- }
- return (exists $results{$test}{'normal'}{'fail'}{$float}
- ? $results{$test}{'normal'}{'fail'}{$float} : "0");
-
-}
-
# Output the defines for a single test
sub output_test {
my ($file, $test, $name) = @_;
my ($ldouble, $double, $float, $ildouble, $idouble, $ifloat);
my ($type);
- # Do we have ulps/failures?
+ # Do we have ulps?
if (!exists $results{$test}{'type'}) {
return;
}
@@ -697,16 +651,6 @@ sub output_test {
$ifloat = &get_ulps ($test, $type, "ifloat");
print $file "#define DELTA$name CHOOSE($ldouble, $double, $float, $ildouble, $idouble, $ifloat)\t/* $test */\n";
}
-
- if (exists $results{$test}{'has_fails'}) {
- $ldouble = &get_failure ($test, "ldouble");
- $double = &get_failure ($test, "double");
- $float = &get_failure ($test, "float");
- $ildouble = &get_failure ($test, "ildouble");
- $idouble = &get_failure ($test, "idouble");
- $ifloat = &get_failure ($test, "ifloat");
- print $file "#define FAIL$name CHOOSE($ldouble, $double, $float $ildouble, $idouble, $ifloat)\t/* $test */\n";
- }
}
# Print include file
diff --git a/math/libm-test.inc b/math/libm-test.inc
index c58bd45..4e1eb72 100644
--- a/math/libm-test.inc
+++ b/math/libm-test.inc
@@ -223,8 +223,6 @@ static int noErrors; /* number of errors */
static int noTests; /* number of tests (without testing exceptions) */
static int noExcTests; /* number of tests for exception flags */
static int noErrnoTests;/* number of tests for errno values */
-static int noXFails; /* number of expected failures. */
-static int noXPasses; /* number of unexpected passes. */
static int verbose;
static int output_max_error; /* Should the maximal errors printed? */
@@ -260,12 +258,6 @@ static FLOAT max_error, real_max_error, imag_max_error;
__imag__ __retval = (imag); \
__retval; })
-#define BUILD_COMPLEX_INT(real, imag) \
- ({ __complex__ int __retval; \
- __real__ __retval = (real); \
- __imag__ __retval = (imag); \
- __retval; })
-
#define MANT_DIG CHOOSE ((LDBL_MANT_DIG-1), (DBL_MANT_DIG-1), (FLT_MANT_DIG-1), \
(LDBL_MANT_DIG-1), (DBL_MANT_DIG-1), (FLT_MANT_DIG-1))
@@ -304,11 +296,11 @@ print_float (FLOAT f)
/* Should the message print to screen? This depends on the verbose flag,
and the test status. */
static int
-print_screen (int ok, int xfail)
+print_screen (int ok)
{
if (output_points
&& (verbose > 1
- || (verbose == 1 && ok == xfail)))
+ || (verbose == 1 && ok == 0)))
return 1;
return 0;
}
@@ -317,25 +309,21 @@ print_screen (int ok, int xfail)
/* Should the message print to screen? This depends on the verbose flag,
and the test status. */
static int
-print_screen_max_error (int ok, int xfail)
+print_screen_max_error (int ok)
{
if (output_max_error
&& (verbose > 1
- || ((verbose == 1) && (ok == xfail))))
+ || ((verbose == 1) && (ok == 0))))
return 1;
return 0;
}
/* Update statistic counters. */
static void
-update_stats (int ok, int xfail)
+update_stats (int ok)
{
++noTests;
- if (ok && xfail)
- ++noXPasses;
- else if (!ok && xfail)
- ++noXFails;
- else if (!ok && !xfail)
+ if (!ok)
++noErrors;
}
@@ -419,7 +407,7 @@ fpstack_test (const char *test_name)
static void
-print_max_error (const char *func_name, FLOAT allowed, int xfail)
+print_max_error (const char *func_name, FLOAT allowed)
{
int ok = 0;
@@ -432,20 +420,19 @@ print_max_error (const char *func_name, FLOAT allowed, int xfail)
print_function_ulps (func_name, max_error);
- if (print_screen_max_error (ok, xfail))
+ if (print_screen_max_error (ok))
{
printf ("Maximal error of `%s'\n", func_name);
printf (" is : %.0" PRINTF_NEXPR " ulp\n", FUNC(ceil) (max_error));
printf (" accepted: %.0" PRINTF_NEXPR " ulp\n", FUNC(ceil) (allowed));
}
- update_stats (ok, xfail);
+ update_stats (ok);
}
static void
-print_complex_max_error (const char *func_name, __complex__ FLOAT allowed,
- __complex__ int xfail)
+print_complex_max_error (const char *func_name, __complex__ FLOAT allowed)
{
int ok = 0;
@@ -461,7 +448,7 @@ print_complex_max_error (const char *func_name, __complex__ FLOAT allowed,
print_complex_function_ulps (func_name, real_max_error, imag_max_error);
- if (print_screen_max_error (ok, xfail))
+ if (print_screen_max_error (ok))
{
printf ("Maximal error of real part of: %s\n", func_name);
printf (" is : %.0" PRINTF_NEXPR " ulp\n",
@@ -475,7 +462,7 @@ print_complex_max_error (const char *func_name, __complex__ FLOAT allowed,
FUNC(ceil) (__imag__ allowed));
}
- update_stats (ok, xfail);
+ update_stats (ok);
}
@@ -493,13 +480,13 @@ test_single_exception (const char *test_name,
{
if (fetestexcept (fe_flag))
{
- if (print_screen (1, 0))
+ if (print_screen (1))
printf ("Pass: %s: Exception \"%s\" set\n", test_name, flag_name);
}
else
{
ok = 0;
- if (print_screen (0, 0))
+ if (print_screen (0))
printf ("Failure: %s: Exception \"%s\" not set\n",
test_name, flag_name);
}
@@ -509,13 +496,13 @@ test_single_exception (const char *test_name,
if (fetestexcept (fe_flag))
{
ok = 0;
- if (print_screen (0, 0))
+ if (print_screen (0))
printf ("Failure: %s: Exception \"%s\" set\n",
test_name, flag_name);
}
else
{
- if (print_screen (1, 0))
+ if (print_screen (1))
printf ("%s: Exception \"%s\" not set\n", test_name,
flag_name);
}
@@ -566,14 +553,14 @@ test_single_errno (const char *test_name, int errno_value,
{
if (errno_value == expected_value)
{
- if (print_screen (1, 0))
+ if (print_screen (1))
printf ("Pass: %s: errno set to %d (%s)\n", test_name, errno_value,
expected_name);
}
else
{
++noErrors;
- if (print_screen (0, 0))
+ if (print_screen (0))
printf ("Failure: %s: errno set to %d, expected %d (%s)\n",
test_name, errno_value, expected_value, expected_name);
}
@@ -595,7 +582,7 @@ test_errno (const char *test_name, int errno_value, int exceptions)
static void
check_float_internal (const char *test_name, FLOAT computed, FLOAT expected,
- FLOAT max_ulp, int xfail, int exceptions,
+ FLOAT max_ulp, int exceptions,
FLOAT *curr_max_error)
{
int ok = 0;
@@ -665,7 +652,7 @@ check_float_internal (const char *test_name, FLOAT computed, FLOAT expected,
}
}
- if (print_screen (ok, xfail))
+ if (print_screen (ok))
{
if (!ok)
printf ("Failure: ");
@@ -683,7 +670,7 @@ check_float_internal (const char *test_name, FLOAT computed, FLOAT expected,
printf (" max.ulp : % .4" PRINTF_NEXPR "\n", max_ulp);
}
}
- update_stats (ok, xfail);
+ update_stats (ok);
fpstack_test (test_name);
errno = 0;
@@ -692,9 +679,9 @@ check_float_internal (const char *test_name, FLOAT computed, FLOAT expected,
static void
check_float (const char *test_name, FLOAT computed, FLOAT expected,
- FLOAT max_ulp, int xfail, int exceptions)
+ FLOAT max_ulp, int exceptions)
{
- check_float_internal (test_name, computed, expected, max_ulp, xfail,
+ check_float_internal (test_name, computed, expected, max_ulp,
exceptions, &max_error);
}
@@ -702,11 +689,10 @@ check_float (const char *test_name, FLOAT computed, FLOAT expected,
static void
check_complex (const char *test_name, __complex__ FLOAT computed,
__complex__ FLOAT expected,
- __complex__ FLOAT max_ulp, __complex__ int xfail,
+ __complex__ FLOAT max_ulp,
int exception)
{
FLOAT part_comp, part_exp, part_max_ulp;
- int part_xfail;
char *str;
if (asprintf (&str, "Real part of: %s", test_name) == -1)
@@ -715,9 +701,8 @@ check_complex (const char *test_name, __complex__ FLOAT computed,
part_comp = __real__ computed;
part_exp = __real__ expected;
part_max_ulp = __real__ max_ulp;
- part_xfail = __real__ xfail;
- check_float_internal (str, part_comp, part_exp, part_max_ulp, part_xfail,
+ check_float_internal (str, part_comp, part_exp, part_max_ulp,
exception, &real_max_error);
free (str);
@@ -727,11 +712,10 @@ check_complex (const char *test_name, __complex__ FLOAT computed,
part_comp = __imag__ computed;
part_exp = __imag__ expected;
part_max_ulp = __imag__ max_ulp;
- part_xfail = __imag__ xfail;
/* Don't check again for exceptions or errno, just pass through the
zero/inf sign test. */
- check_float_internal (str, part_comp, part_exp, part_max_ulp, part_xfail,
+ check_float_internal (str, part_comp, part_exp, part_max_ulp,
exception & IGNORE_ZERO_INF_SIGN,
&imag_max_error);
free (str);
@@ -741,7 +725,7 @@ check_complex (const char *test_name, __complex__ FLOAT computed,
/* Check that computed and expected values are equal (int values). */
static void
check_int (const char *test_name, int computed, int expected, int max_ulp,
- int xfail, int exceptions)
+ int exceptions)
{
int diff = computed - expected;
int ok = 0;
@@ -756,7 +740,7 @@ check_int (const char *test_name, int computed, int expected, int max_ulp,
if (!ok)
print_ulps (test_name, diff);
- if (print_screen (ok, xfail))
+ if (print_screen (ok))
{
if (!ok)
printf ("Failure: ");
@@ -766,7 +750,7 @@ check_int (const char *test_name, int computed, int expected, int max_ulp,
printf (" should be: %d\n", expected);
}
- update_stats (ok, xfail);
+ update_stats (ok);
fpstack_test (test_name);
errno = 0;
}
@@ -775,7 +759,7 @@ check_int (const char *test_name, int computed, int expected, int max_ulp,
/* Check that computed and expected values are equal (long int values). */
static void
check_long (const char *test_name, long int computed, long int expected,
- long int max_ulp, int xfail, int exceptions)
+ long int max_ulp, int exceptions)
{
long int diff = computed - expected;
int ok = 0;
@@ -790,7 +774,7 @@ check_long (const char *test_name, long int computed, long int expected,
if (!ok)
print_ulps (test_name, diff);
- if (print_screen (ok, xfail))
+ if (print_screen (ok))
{
if (!ok)
printf ("Failure: ");
@@ -800,7 +784,7 @@ check_long (const char *test_name, long int computed, long int expected,
printf (" should be: %ld\n", expected);
}
- update_stats (ok, xfail);
+ update_stats (ok);
fpstack_test (test_name);
errno = 0;
}
@@ -809,7 +793,7 @@ check_long (const char *test_name, long int computed, long int expected,
/* Check that computed value is true/false. */
static void
check_bool (const char *test_name, int computed, int expected,
- long int max_ulp, int xfail, int exceptions)
+ long int max_ulp, int exceptions)
{
int ok = 0;
int errno_value = errno;
@@ -820,7 +804,7 @@ check_bool (const char *test_name, int computed, int expected,
if ((computed == 0) == (expected == 0))
ok = 1;
- if (print_screen (ok, xfail))
+ if (print_screen (ok))
{
if (!ok)
printf ("Failure: ");
@@ -830,7 +814,7 @@ check_bool (const char *test_name, int computed, int expected,
printf (" should be: %d\n", expected);
}
- update_stats (ok, xfail);
+ update_stats (ok);
fpstack_test (test_name);
errno = 0;
}
@@ -840,7 +824,7 @@ check_bool (const char *test_name, int computed, int expected,
static void
check_longlong (const char *test_name, long long int computed,
long long int expected,
- long long int max_ulp, int xfail,
+ long long int max_ulp,
int exceptions)
{
long long int diff = computed - expected;
@@ -856,7 +840,7 @@ check_longlong (const char *test_name, long long int computed,
if (!ok)
print_ulps (test_name, diff);
- if (print_screen (ok, xfail))
+ if (print_screen (ok))
{
if (!ok)
printf ("Failure:");
@@ -866,7 +850,7 @@ check_longlong (const char *test_name, long long int computed,
printf (" should be: %lld\n", expected);
}
- update_stats (ok, xfail);
+ update_stats (ok);
fpstack_test (test_name);
errno = 0;
}
@@ -14186,10 +14170,6 @@ main (int argc, char **argv)
printf (" %d test cases plus %d tests for exception flags and\n"
" %d tests for errno executed.\n",
noTests, noExcTests, noErrnoTests);
- if (noXFails)
- printf (" %d expected failures occurred.\n", noXFails);
- if (noXPasses)
- printf (" %d unexpected passes occurred.\n", noXPasses);
if (noErrors)
{
printf (" %d errors occurred.\n", noErrors);
--
Joseph S. Myers
joseph@codesourcery.com