Re: testsuites classification

[Joel Sherrill <joel dot sherrill at OARcorp dot com>]

Daniel Berlin wrote:
> 
> "GNU Team" <gnuteam@kpit.com> writes:
> 
> > I have a question regarding gcc testsuites.
> > How do I categorize them ?
> > After running the gcc 3.0 test suite on  gcc 2.95.3 (sh-coff-gcc)
> > I got following results.
> >
> > GCC   PASS          10077
> >       FAIL          4927
> >       XPASS         2
> >       XFAIL         78
> >       UNRESOLVED      132
> >       UNSUPPORTED     59
> >       WARNINGS        23
> >       ERRORS  3
> >
> >
> > Now, is there any document which classifies these testsuites ?
> > ie. which tests must PASS and which may not pass ?
> > Or does it mean that compiler has 4500 bugs which must be fixed ?
> It means the compiler is failing 4900 tests.
> However, this is not an indication of the number of bugs.
> For the c-torture tests, for instance, unless the test specifies
> otherwise, each test is compiled 8 times (with different flags each
> time).
> Other than that, all tests that aren't XFAIL, should be passing (in an
> ideal world).

I recall that many of the sh-coff tests on gcc-30 from CVS are failing
due to linker errors.  You really should look at the file 
gcc/testsuite/gcc.log for more details on the failures.  My experience
is that when the number is this high, they tend to cluster.  For example,
last week one VERY minor problem caused about 3600 failures for sparc-elf.

> >
> >
> > Arati.
> 
> --
> "I'm so hyper...  (Said with a very dull voice.)
> "-Steven Wright
> 
> ------
> Want more information?  See the CrossGCC FAQ, http://www.objsw.com/CrossGCC/
> Want to unsubscribe? Send a note to crossgcc-unsubscribe@sourceware.cygnus.com

-- 
Joel Sherrill, Ph.D.             Director of Research & Development
joel@OARcorp.com                 On-Line Applications Research
Ask me about RTEMS: a free RTOS  Huntsville AL 35805
Support Available                (256) 722-9985

------
Want more information?  See the CrossGCC FAQ, http://www.objsw.com/CrossGCC/
Want to unsubscribe? Send a note to crossgcc-unsubscribe@sourceware.cygnus.com