This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: towards zero FAIL for make installcheck


On Thu, 2011-12-01 at 10:41 -0500, Dave Brolley wrote:
> On 12/01/2011 06:57 AM, Mark Wielaard wrote:
> > As you can see the low number of FAILs (unexpected failures) is
> > compensated by a high number of KFAILs (known failures) and UNTESTED
> The recent spike in the number of UNTESTED was me reducing the number of 
> tests run by the unprivileged_embedded_C.exp test. I simply changed it 
> to test on a fraction of the tapset functions in our library and to 
> report the rest as UNTESTED. The test still covers all variants of no 
> embedded C/unprivileged/myproc-unprivileged.
> 
> It was an arbitrary decision on my part to report the skipped functions 
> as UNTESTED. If it is preferred, I could change it to simply ignore them.

How exactly "would be nice to run" tests fit it in is somewhat unclear.
In theory the difference between UNTESTED and UNSUPPORTED is that the
first could be made to work (but some setup or an earlier test failure
prevented it) and the later means the test never can be run successfully
on this setup/arch. But there is no clear indicator for "EXPECTED
UNTESTED" (because the user said so). I think ignoring them is slightly
more helpful, then you can more easily see how many "real" UNTESTED
tests there were.

Thanks,

Mark


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]