RE: KPASS in test results

["Stone, Joshua I" <joshua dot i dot stone at intel dot com>]

On Friday, January 19, 2007 8:42 AM, Mike Mason wrote:
> I also figured out that 9999 represents a bug # and *-*-* is the
> targetname (I think). I don't know yet where targetname on a specific
> machine is defined.  

I'm not sure where it's defined, but you can see the config triplet for
your system at the beginning of the run, something like:

  Native configuration is i686-redhat-linux-gnu

> - When is it appropriate to remove a setup_kfail line?

When the problem is believed to be fixed.

> - If we're going to use setup_kfail, shouldn't we be filling in the
> bug # and target variables? 

Yes, I think we should have bugs open for all of these, else we'll just
end up ignoring the failure.

As for the target variable, it's not always easy to mask the known
failure appropriately.  If it's architecture specific, you can, but if
the problem is specific to a range of kernels (e.g. fails on RHEL4 but
works on FC6) I don't think you can differentiate.  However, we might be
able to set those up to only call setup_kfail after checking uname.

> - Is it wise to even use setup_kfail?  It might mask failures that
> should be getting more attention. I realize the intent of setup_kfail
> is to distinguish known bugs from new bugs, but should we really do
> that?   

I think it's fine as long as there's a bug for it -- and if bugs aren't
being addressed that's a separate issue we can deal with.  It doesn't
seem right to setup_kfail with a dummy bug number though (9999), because
then we really are just ignoring it.

The only exception is if there's some test that we know fails on a
particular setup, which we don't intend to fix or are unable to fix.  In
that case, the test itself should be written to return "unsupported" on
that setup rather than failure.


Josh