RE: testsuites classification

[David Korn <dkorn at pixelpower dot com>]

>-----Original Message-----
>From: Joel Sherrill [mailto:joel.sherrill@OARcorp.com]
>Sent: 06 August 2001 22:00
>To: Daniel Berlin
>Cc: gnuteam@kpit.com; gcc-bugs@gcc.gnu.org;
>crossgcc@sourceware.cygnus.com
>Subject: Re: testsuites classification
>
>
>
>
>Daniel Berlin wrote:
>> 
>> "GNU Team" <gnuteam@kpit.com> writes:
>> 
>> > I have a question regarding gcc testsuites.
>> > How do I categorize them ?
>> > After running the gcc 3.0 test suite on  gcc 2.95.3 (sh-coff-gcc)
>> > I got following results.
>> >
>> > GCC   PASS          10077
>> >       FAIL          4927
>> >       XPASS         2
>> >       XFAIL         78
>> >       UNRESOLVED      132
>> >       UNSUPPORTED     59
>> >       WARNINGS        23
>> >       ERRORS  3


  Since you're using the 3.0 testsuite with 2.95.3, an awful lot of these
failures will be down to the C99 language features that were added in 3.0;
the tests for these are bound to fail on 2.95 since that was pre-C99.  So
there's an awful lot of those FAILs that should really be expected.  I don't
know precisely which tests cover the C99 syntax but if they all lived in
a subdirectory of the testsuite you could easily remove them.


        DaveK
-- 
"Computer games don't affect kids; I mean if Pac-Man affected us as kids,
we'd all be running around in darkened rooms, munching magic pills and
listening to repetitive electronic music."
    - Kristian Wilson, Nintendo, Inc, 1989.

 

>> >
>> >
>> > Now, is there any document which classifies these testsuites ?
>> > ie. which tests must PASS and which may not pass ?
>> > Or does it mean that compiler has 4500 bugs which must be fixed ?
>> It means the compiler is failing 4900 tests.
>> However, this is not an indication of the number of bugs.
>> For the c-torture tests, for instance, unless the test specifies
>> otherwise, each test is compiled 8 times (with different flags each
>> time).
>> Other than that, all tests that aren't XFAIL, should be 
>passing (in an
>> ideal world).
>
>I recall that many of the sh-coff tests on gcc-30 from CVS are failing
>due to linker errors.  You really should look at the file 
>gcc/testsuite/gcc.log for more details on the failures.  My experience
>is that when the number is this high, they tend to cluster.  
>For example,
>last week one VERY minor problem caused about 3600 failures 
>for sparc-elf.
>
>> >
>> >
>> > Arati.
>> 
>> --
>> "I'm so hyper...  (Said with a very dull voice.)
>> "-Steven Wright
>> 
>> ------
>> Want more information?  See the CrossGCC FAQ, 
>http://www.objsw.com/CrossGCC/
>> Want to unsubscribe? Send a 
>note to crossgcc-unsubscribe@sourceware.cygnus.com
>
>-- 
>Joel Sherrill, Ph.D.             Director of Research & Development
>joel@OARcorp.com                 On-Line Applications Research
>Ask me about RTEMS: a free RTOS  Huntsville AL 35805
>Support Available                (256) 722-9985
>
>------
>Want more information?  See the CrossGCC FAQ, 
http://www.objsw.com/CrossGCC/
Want to unsubscribe? Send a note to
crossgcc-unsubscribe@sourceware.cygnus.com


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

------
Want more information?  See the CrossGCC FAQ, http://www.objsw.com/CrossGCC/
Want to unsubscribe? Send a note to crossgcc-unsubscribe@sourceware.cygnus.com