This is the mail archive of the
mailing list for the Mauve project.
Re: Mauve wishlist
David Gilbert wrote:
1) constant number of tests, regardless of exceptions being thrown or
Mauve does have a design flaw where it can be tricky to automatically
assign a unique identifier to each check(), and this makes it hard to
compare two Mauve runs (say a test of the latest Classpath CVS vs the
last release, or the Classpath vs JDK 1.5 - both of which would be
Right. We all understand the problem - its just the solution is what we
need to agree on :)
I think the absolute number is meaningless however you count the
tests, so I don't see this as an advantage.
Yes, numbers alone are meaningless, but with the current design, all the
results are meaningless without a lot of context. The real issue is
having a simple way to uniquely identify each test case for the purpose
of identifying regressions. This becomes fundamentally much easier when
1 test() method corresponds to one test case.
It is not reasonable to expect test case developers ensure that all
tests "run linearly". Exceptions can potentially be thrown at any time,
so to ensure linearity, every check() call would need wrapped with a
You'll lose the ability to distinguish between an existing failure
where (say) 1 out of 72 checks fail, and after some clever patch 43
out of 72 checks fail, but the new system reports both as 1 test failure.
This is a valid concern. However, we will still track exactly which
check() calls fail so that in the event of a test failure, a full
diagnostic can be provided. In addition, we can still count the total
number of check() calls executed, for statistical purposes.
If the reduced test-case granularity does prove to be problematic in
some cases - say some test() where a small number of checks fail for
cases that are "hard to fix" problems and thus should be "xfailed" (to
use gcc/dejagnu lingo), then that test case should probably be split.
Alternatively, to avoid splitting things across multiple test classes,
we could add JUnit-style support for multiple test() methods in a single