This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH roland/test-unsupported] Let tests result in UNSUPPORTED; use that for unbuildable C++ cases
- From: Joseph Myers <joseph at codesourcery dot com>
- To: Roland McGrath <roland at hack dot frob dot com>
- Cc: "GNU C. Library" <libc-alpha at sourceware dot org>
- Date: Sat, 7 Mar 2015 01:37:08 +0000
- Subject: Re: [PATCH roland/test-unsupported] Let tests result in UNSUPPORTED; use that for unbuildable C++ cases
- Authentication-results: sourceware.org; auth=none
- References: <20150307010200 dot B51122C3B39 at topped-with-meat dot com>
On Fri, 6 Mar 2015, Roland McGrath wrote:
> evaluate-test.sh now follows the Autotest convention of exit code 77
> meaning a test that cannot be built or cannot be usefully run,
> i.e. UNSUPPORTED.
I wonder if any existing tests return a count of errors (incrementing a
variable on error rather than setting it to 1, then returning that
variable) and could potentially return 77 by accident. But such tests
would already be broken if they returned 256, so any issues can be
considered pre-existing.
> We have many existing cases of tests that do compile but are turned into
> no-ops that always pass either at compile time or at run time. Once this
> lands, we should start converting those such that they report UNSUPPORTED
> rather that PASS. But that is a later step.
And if cross-compiling without test-wrapper (so run-built-tests = no), in
principle all affected tests should be UNSUPPORTED. And at some point
there's the issue of allowing tests to report status for their individual
assertions (so a test might have some assertions that pass and some that
are unsupported).
> Notably, I've changed the summarizing stuff so that it prints out all
> results that are neither PASS nor XFAIL, so that catches all ERROR and
> XPASS as well as the new UNSUPPORTED. Similarly, I've changed the logic
> that decides whether 'make check' as a whole succeeded or not so that it
> rejects anything that is not PASS, XFAIL, or UNSUPPORTED, rather than
> rejecting only things that are exactly FAIL or ERROR. This means that any
> existing XPASS cases now cause 'make check' to fail. I think that's the
> right thing, since XPASS means something funny is going on and either the
> test needs to be changed or the XFAIL expectation needs to be dropped. I
> also think it's definitely the right thing to both display and barf on any
> statuses that are not in the expected set.
I don't think XPASSes should cause "make check" to fail - or at least,
that's contrary to how we use XFAILs at present. XPASSes appear right now
for cases where the expected failure depends on conditions there is
otherwise no need to test for in configure or makefiles. In conform/,
that includes (a) tests requiring GCC 4.7 or later, (b) tests that may
depend on whether some non-glibc package has installed ndbm.h, (c) tests
that are expected to fail on at least one of x86_64 and x86 but where the
failure is for an architecture-specific (library or header) issue and so
the tests pass on some other platforms. Only (c) is a case where the
XFAILs could simply be moved to sysdeps if desired and the relevant
sysdeps condition is worked out. It's not clear to me that it's useful to
condition these XFAILs more precisely. And XFAILs might well be useful
for a few nptl/ and rt/ tests that are known to fail intermittently,
possibly because of Linux kernel issues (where such a test issue is known,
it seems better to me for the test to be variably XPASS or XFAIL, than
variably PASS or FAIL).
--
Joseph S. Myers
joseph@codesourcery.com