This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: Do not terminate default test runs on test failure
- From: "Joseph S. Myers" <joseph at codesourcery dot com>
- To: Andreas Schwab <schwab at suse dot de>
- Cc: Carlos O'Donell <carlos at redhat dot com>, <libc-alpha at sourceware dot org>
- Date: Tue, 1 Apr 2014 14:27:48 +0000
- Subject: Re: Do not terminate default test runs on test failure
- Authentication-results: sourceware.org; auth=none
- References: <Pine dot LNX dot 4 dot 64 dot 1403071741020 dot 6302 at digraph dot polyomino dot org dot uk> <mvm38hyfl2x dot fsf at hawking dot suse dot de> <Pine dot LNX dot 4 dot 64 dot 1403311712520 dot 15146 at digraph dot polyomino dot org dot uk> <87ha6e3yx3 dot fsf at igel dot home> <Pine dot LNX dot 4 dot 64 dot 1403312023250 dot 10829 at digraph dot polyomino dot org dot uk> <mvmppl1ebt8 dot fsf at hawking dot suse dot de> <Pine dot LNX dot 4 dot 64 dot 1404011146310 dot 24927 at digraph dot polyomino dot org dot uk> <mvmppl19ox7 dot fsf at hawking dot suse dot de> <Pine dot LNX dot 4 dot 64 dot 1404011405540 dot 24927 at digraph dot polyomino dot org dot uk> <mvmioqt9lpq dot fsf at hawking dot suse dot de>
On Tue, 1 Apr 2014, Andreas Schwab wrote:
> "Joseph S. Myers" <joseph@codesourcery.com> writes:
>
> > Use cases that involve identifying a list of failures from a build log are
> > expected to involve changes to the system you use to list failures - that
> > is, this is working as designed.
>
> It is unacceptable that there is no summary just because one of the test
> cases fails to build, since there is no other way to see which test
> cases have failed if the only source of information is the build log.
> This is never a problem with dejagnu, for example.
It is the case with DejaGnu. The equivalent in DejaGnu is that a problem
in a .exp file cause an ERROR that terminates the test run early or
otherwise prevents some tests from being run (ERRORs in DejaGnu can't be
reliably associated with individual tests). I've seen that plenty of
times.
The output of "make" is not a stable interface (the cases in which the
exit status is nonzero are such an interface). If you have automation to
list failures from the output of "make", it needs changing. If you want
to use -k (and I discourage using -k with "make check" after my changes)
and allow build failures, either use stop-on-test-failure and otherwise
keep your old automation, or combine failures from the make logs with a
concatentation of all .test-result files from the tree where you ran "make
-k check".
I don't object to counting the builds of testcases as tests themselves and
using $(evaluate-test) in the makefile rules (obviously you need to avoid
conflicts in the PASS/FAIL lines between the entry for the build and the
entry for the execution), but that would be a further incremental
improvement to remove more cases for use of -k or external automation to
identify failures.
--
Joseph S. Myers
joseph@codesourcery.com