Re: [testsuite] add gdb.cp/gdb1355.exp

[David Carlton <carlton at kealia dot com>]

On Wed, 17 Sep 2003 20:56:12 -0400, Daniel Jacobowitz <drow@mvista.com> said:
> On Wed, Sep 17, 2003 at 08:53:57PM -0400, Michael Elizabeth Chastain wrote:

>>> For one thing, it would be an XFAIL, because it's a GCC bug,
>>> not a GDB bug.

>> The test script, gdb.cp/gdb1355.exp, refers to PR gdb/1355.
>> gdb/1355 is an external PR and it refers to PR gcc/12066.
>> So there is a gdb PR in there.

>>> For another thing, though, the bug in question has been fixed,
>>> so we don't expect it to fail: if it does, it should show up as a FAIL.

>> This has been a controversy in the past, too.

>> My view is that "KFAIL" means "Known FAIL", which basically means
>> there is a PR for it (the PR is the locus of knowledge).

> I don't think that was the consensus.  KFAILs are known failures of
> the tool under test, i.e. bugs in it.  This is a problem in GDB's
> input.  That makes it an xfail.

That's my understanding as well.  See, for example,
<http://sources.redhat.com/ml/gdb/2002-12/msg00193.html>: it says,
among other things, "An XFAIL means that we have found a bug in
software external to gdb" and "KFAIL means that the failure is due to
a known bug in gdb".  A short thread followed that message; nobody
disagreed with those statements.  Also, in the thread entitled "GDB
`cannotfix' pr state, require PR with xfail `moving forward'.", it
seems to me that there was consensus that new xfails would be
identified with a PR, and that that PR should come from GDB's bug
database.  (See, e.g.,
<http://sources.redhat.com/ml/gdb/2003-01/msg00324.html>.)  So that,
too, argues against "GDB bug database implies KFAIL".

>>> I would leave in the new test, with branches and comments as is,
>>> but I would change all the occurrences of kfail to fail.
>> 
>> I prefer gdb1355.exp the way it is but I would be okay with that change
>> if other people want it that way.

I can't find a mailing list reference for this, so I'll repeat my
argument for changing KFAIL to FAIL.  Let's say that we have an
annoying GDB bug.  We add a test to the suite reflecting the bug;
initially, we've KFAILed the bug.  But then somebody fixes the bug.
If the bug is left as a KFAIL, instead of changing to FAIL, then we
have problems in these two scenarios:

Scenario 1: The bug is fixed.  A year later, the bug in question
regresses.  But, if you just run 'make check', this isn't so obvious:
no message gets printed in the output, and while the numbers at the
end of the output will change, those numbers fluctuate for lots of
reasons.  So it will only be noticed by somebody who actually looks at
gdb.sum (or does the moral equivalent); ideally that will happen, but
I'd rather not count on it (especially if the bug is on an obscure
platform).

Scenario 2: The fix works on platform A, but not on platform B.  And
nobody using platform B has been paying close enough attention to the
discussion of the bug to realize that the test is supposed to start
passing.  Then, nobody might ever realize that something's wrong:
platform A users think that everything is fine (which is the case for
them), and platform B have no indication from the testsuite that
something is wrong and that the patch didn't work as indicated.


I'm especially worried about Scenario 2, but I don't like Scenario 1
either.  And I don't see any correspondingly strong scenarios that
argue in the other direction.  About the only problem with changing it
from a KFAIL to a FAIL is that, if the bug reoccurs, you won't learn
from make check (or gdb.sum or gdb.log) that the bug in question is
one that has been seen earlier.  But that's not a big deal, since you
know which test failed, so as long as that test has a comment in it
saying "this was PR gdb/123, but it's been fixed now", then you'll
have access to the information you need.  Which is why, in the
situation at question, I like it that you have the explicit failure
branches, containing an informative comment as to when the bug
occurred - I just want the letter 'k' removed in a few places. :-)

David Carlton
carlton@kealia.com