While I agree in principle with the things you quote below, and in
general with most of the junit FAQ (as I have for many years), I do also
believe that we shouldn't necessarily blindly follow whatever is stated
in the FAQ. I mean that in very generic terms: the purpose of having a
testsuite is to provide us with clear, consistent, and above all useful
results.
One thing I have always had an issue with concerning junit is the
blending of ERROR and FAIL in junit 4. I believe that the distinction
between a failed test (assertions failed or expected failure mode (e.g.
exception) occcured) and a test execution that failed (unexpected event
caused the test to not complete as expected, i.e. something made it
impossible to make a correct PASS/FAIL determination). In that sense, I
do not think it to be wise to always take the strict rule of just
letting exceptions bubble up to the framework. To me, the most
important word in the FAQ about this topic is "unexpected". If there is
an exception that is known to validly occur in a failure scenario (it
makes the test fail to satisfy it assertions) then that should be
reflected in the result differently from an unexpected exception that
interferes with the test execution.
There are obviously different schools of thought, but given that Frysk
is already using 2 testing frameworks that are set up to return results
that fall within the POSIX categories (as described in the dejagnu
documentation as reference - discussed in past months), it would make
sense that we make use of the full spectrum of result messages in a
meaningful way.