In followup to our conversation on the conference call... The summary file
does not capture the ERROR messages from runtest, and those are significant
in the processing of test results.
Whenm e.g. the staprun executable is not setuid root (or equiv privs), some
tests print out an ERROR about this, but the test result is (for some tests)
actually reported as XFAIL. Now, per the discussion on the call, that should
be interpreted as a PASS. So... the summary file ends up listing a test
result that will get reported as a PASS when it most definitely was not a
successful test execution.
That kind of cases can be resolved based on the verbose log.
Another problem (brought to light by a question from David Wilder) is that
the dejagnu summary output seems to be inconsistent with the actual log
messages (both in the summary and verbose logs):
stap Grep
---- ----
PASS (expected passes) 463 462
FAIL (unexpected failures) 19 19
XFAIL (expected failures) 152 152
XPASS (unknown successes) 1 0
KFAIL (known failures 5 5
UNTESTED (untested) 26 26
UNSUPOPRTED (unsupported) 2 2
As you can see, dejagnu is reporting 1 extra PASS, and 1 extra XPASS, yet
a grep on the log cannot find those two entries. Does anyone have any idea
where they are coming from?
As to the potential for non-public information making its way into the verbose
log... If and when that occurs, I am certain that it would be treated as a
high priority bug in systemtap or its testsuite, and that it would be resolved
rather quickly. For example, the samba build (and test) farm has a long
history of doing this kind of stuff (and they have had their share of somewhat
unexpected things happening). The main thing usually is to ensure that all
testing is done in a sufficiently safe environment, both in terms of not
letting anything leak out that shouldn't, and in terms of ensuring that a
misbehaving test won't cause more damage.
Cheers,