Re: RFC: Known Failures [Was: RFD: Testsuite cases for inferior function calls]

[Fernando Nasser <fnasser at cygnus dot com>]

"Peter.Schauer" wrote:
> 
> > "Peter.Schauer" wrote:
> > >
> > > KFAILs would be fine with me, as long as they show up in the testsuite run
> > > output.
> > >
> >
> > Note that *all* results go to the testsuite run output.  You are referring to the
> > summary file, perhaps.
> 
> No, I meant the standard output of a testsuite run. I can quickly see any
> FAILs there without grepping etc., and I think it only fair to expose them
> to any installer of GDB who bothers to run the testsuite.
> 

That may be a question of taste or opinion.  I definitively don't want to see
anything more than FAILs on the stdout.  Maybe we can vote on this, or make it
an option switch.

> And so I would like to appear the KFAILs there as well, otherwise I fear
> that KFAILs will be forgotten just like XFAILs.
>

No they won't because they are know to be defects and there will be entries in
the bug database corresponding to each one.  That is a condition to mark something
as a known failure -- otherwise it would be UNknown failure :-)
 
> There used to be a time when testcases were required for every submitted
> change.
> Gathering from recent postings it seems like you might not be allowed to add
> new testcases if they fail on any conceivable platform :-).
> 

That is not what I said.  We are talking about tests that a known to generally
fail and to pass only on a few targets (3, I believe) that have been fixed.

I did not even said about not checking it in, but to xfail them to the targets
that have not yet been fixed (and for which we have no precise plan about when
we will be able to do it).


> I don't have very strong opinions about FAIL/XFAIl/KFAIL, but I'd really
> like to see a clear strategy for testcases, perhaps written down somewhere.
> 

Yes, that should be nice.  I believe Andrew can add that to the rules once a
decision is made.


> > It is not a release problem.  It is an engineering problem: we want to be able to
> > clearly know if a change caused regressions, without having to memorize which is
> > the usual level of FAILs for each host-x-target combination.
> 
> What is the problem with a diff against a last `good' version of gdb.sum
> or the testsuite output ?
> 
> Locally I am even diffing a processed version of gdb.log against a known
> version, because it uncovers many subtle problems (e.g new GDB warning
> messages, missing prompt consumption etc.).
> 

It may not always be available, it complicates visualizing data about failures
on a table, etc.  We want to automate the tracking of breakage of the numerous
combinations of hosts and targets.


-- 
Fernando Nasser
Red Hat - Toronto                       E-Mail:  fnasser@cygnus.com