This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch
On Mon, 17 Sep 2012, Tomas Dohnalek wrote:
> Ok, but then the behaviour of running simple `make check' (no -k) will be
> changed, is that ok? I guess that it is probably desired to run the whole
> test-suite even when error occurs, but maybe someone has some objections?
Good point - I suppose you need to check MAKEFLAGS in the test rule and
emulate -k handling there, rather than just unconditionally ignoring
errors from $(MAKE) tests, so that errors from the inner make are properly
ignored or not depending on whether -k is passed.
Ideally I think the whole testsuite should run in any case (meaning
passing down -k whether or not the user specified it) and "make check"
should exit with a nonzero status if any test had an *unexpected* failure
(including failure of any of the commands involved in building an test, or
the various separate makefile rules that do bits of setup for running
tests) - printing on stdout/stderr a summary of the unexpected failures.
But that would be for a later stage, since you'd need to develop a way to
track results of test compilation / setup as well as the tests themselves,
and a way for expectedness of failures to be more directly visible to the
test harness. (As is, expected failures simply use "-" in their makefile
rules directly, e.g. in posix/Makefile for $(objpfx)annexc.out.)
--
Joseph S. Myers
joseph@codesourcery.com