This is the mail archive of the newlib@sources.redhat.com mailing list for the newlib project.
Index Nav: | [Date Index] [Subject Index] [Author Index] [Thread Index] | |
---|---|---|
Message Nav: | [Date Prev] [Date Next] | [Thread Prev] [Thread Next] |
Other format: | [Raw text] |
When integrating dejagnu-based testsuites such as newlib's into an automatic regression testing system, the PASS and FAIL lines from the ".sum" files from successive runs are extracted and compared. This requires the test assertion names after PASS or FAIL to be unique (within the tests for each multilib; successive multilibs repeat the same names) and stable so that the differences show only genuine changes, not changes because of (for example) the date in a source directory name changing. In addition, it's desirable for a test failure not to change the set of test assertion names, so if a compilation test fails the following execution test should be marked UNRESOLVED rather than disappearing altogether from the test output.
At present, newlib.sum can have lines such as:
FAIL: Failed to compile /scratch/gcc/nightly-2005-07-04-csl-arm/src/newlib-mainline/newlib/testsuite/newlib.wctype/twctype.c.
which has two problems: the test assertion name includes the full source directory path, and such a line will only appear if the test fails; if it passes, it won't become (and wouldn't make sense to become):
PASS: Failed to compile /scratch/gcc/nightly-2005-07-04-csl-arm/src/newlib-mainline/newlib/testsuite/newlib.wctype/twctype.c.
Test names such as "newlib.wctype/twctype.c compilation", which can appear after either PASS or FAIL, avoid these problems. This patch accordingly changes the newlib testsuite to use such test names, with additional information about unexpected results from output comparison going in "verbose -log" lines in the log rather than in the FAIL name. Tested by running the testsuite for arm-none-eabi. OK to commit?
2005-07-06 Joseph S. Myers <joseph@codesourcery.com>
* testsuite/lib/checkoutput.exp (newlib_check_output): Use test names after PASS and FAIL which do not depend on source directory name or on whether test passed or failed. * testsuite/lib/passfail.exp (newlib_pass_fail): Likewise.
diff -rupN testsuite.orig/lib/checkoutput.exp testsuite/lib/checkoutput.exp
--- testsuite.orig/lib/checkoutput.exp 2005-07-05 00:10:08.000000000 +0000
+++ testsuite/lib/checkoutput.exp 2005-07-06 22:39:13.000000000 +0000
@@ -17,9 +17,11 @@ proc newlib_check_output { srcfile expec
set comp_output [newlib_target_compile "$srcfullname" "$test_driver" "executable" ""]
if { $comp_output != "" } {
- fail "Failed to compile $srcfile.\n"
+ fail "$subdir/$srcfile compilation"
+ unresolved "$subdir/$srcfile output"
return
}
+ pass "$subdir/$srcfile compilation"
set result [newlib_load $test_driver ""]
set status [lindex $result 0]
@@ -30,11 +32,12 @@ proc newlib_check_output { srcfile expec
foreach { expectedval } $expectlist {
set gotval [string trim [lindex $output_lines 0] "\r"]
if { ! [string match $expectedval $gotval] } {
- fail "$srcfile: Expected: $expectedval Got: $gotval "
+ verbose -log "$subdir/$srcfile: Expected: $expectedval Got: $gotval "
+ fail "$subdir/$srcfile output"
return
}
set output_lines [lrange $output_lines 1 end]
}
- pass $srcfile
+ pass "$subdir/$srcfile output"
}
diff -rupN testsuite.orig/lib/passfail.exp testsuite/lib/passfail.exp
--- testsuite.orig/lib/passfail.exp 2005-07-05 00:10:08.000000000 +0000
+++ testsuite/lib/passfail.exp 2005-07-06 22:40:09.000000000 +0000
@@ -41,10 +41,12 @@ proc newlib_pass_fail { srcfile } {
set comp_output [newlib_target_compile "$fullsrcfile" "$test_driver" "executable" ""]
if { $comp_output != "" } {
- fail "Failed to compile $fullsrcfile.\n"
+ fail "$subdir/$srcfile compilation"
+ unresolved "$subdir/$srcfile execution"
} else {
+ pass "$subdir/$srcfile compilation"
set result [newlib_load $test_driver ""]
set status [lindex $result 0]
- $status "$fullsrcfile"
+ $status "$subdir/$srcfile execution"
}
}
Index Nav: | [Date Index] [Subject Index] [Author Index] [Thread Index] | |
---|---|---|
Message Nav: | [Date Prev] [Date Next] | [Thread Prev] [Thread Next] |