This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [2.20] [3/6] Support expected failures in .test-result files
- From: "Joseph S. Myers" <joseph at codesourcery dot com>
- To: Carlos O'Donell <carlos at redhat dot com>
- Cc: Brooks Moses <bmoses at google dot com>, libc-alpha <libc-alpha at sourceware dot org>
- Date: Fri, 14 Feb 2014 16:59:03 +0000
- Subject: Re: [2.20] [3/6] Support expected failures in .test-result files
- Authentication-results: sourceware.org; auth=none
- References: <Pine dot LNX dot 4 dot 64 dot 1401100208000 dot 9412 at digraph dot polyomino dot org dot uk> <Pine dot LNX dot 4 dot 64 dot 1401100211530 dot 9412 at digraph dot polyomino dot org dot uk> <52F95BE2 dot 3070004 at google dot com> <Pine dot LNX dot 4 dot 64 dot 1402102315100 dot 26591 at digraph dot polyomino dot org dot uk> <CAOxa4Kpwqt3D6M9D102KvoN_rq1e8AhFpzdO6RDoC=2-SmM29A at mail dot gmail dot com> <52FD9FBB dot 4030801 at redhat dot com> <Pine dot LNX dot 4 dot 64 dot 1402141322120 dot 643 at digraph dot polyomino dot org dot uk> <52FE3577 dot 8030001 at redhat dot com>
On Fri, 14 Feb 2014, Carlos O'Donell wrote:
> I agree with that. However, my worry is that this will lead to an overly
> complex meta-language to describe when XFAILs are expected. If instead a
The meta-language is GNU make, controlling makefile variable settings
(together with arbitrary C code when it's actually the test itself
disabling something). It may not be ideal, but it's what we have at
present to control lots of things about what tests are run and how they
are run.
> It's the "whatever the relevant conditions are" that worries me. The result
> of such language naturally leads me to believe we'll have lots and lots of
> XFAILs with complex conditions because we support many versions of gcc,
> many versions of the kernel, and other related tools. Worse the version
> numbers don't mean the same to all distributions which apply custom patches.
In the kernel case, it will probably be conditionals within the tests,
like the tst-setgetname.c case you dealt with. We also use those for
other cases, e.g. math-tests.h.
Ideally tests would be able to give more fine-grained information in such
cases - as well as an overall result from the test (a dummy PASS from the
exit status here) there would be results from individual assertions within
the test (all UNRESOLVED because the old kernel prevents them from being
tested). But in the absence of that, we can still incrementally stop
tests giving a spurious FAIL when actually they are indicating an issue
external to glibc, or some other known issue. (And incrementally making
more tests use test-skeleton.c would likely help with any future move to
fine-grained test assertion status, in that test-skeleton.c is the natural
place for functions to say "record this assertion as having this result".)
> In this case might we remove old XFAILS if say the kernel version condition
> they depended upon is no longer supported?
Yes.
--
Joseph S. Myers
joseph@codesourcery.com