Re: New Harness, version 1

[Thomas Fitzsimmons <fitzsim at redhat dot com>]

Hi,

On Thu, 2006-04-06 at 10:17 +0100, David Gilbert wrote:
> Anthony Balkissoon wrote:
> 
> >Okay, the new Harness is checked in, please use it and comment on
> >features you would like added.  
> >
> >Running tests is now much easier, please use the harness script in the
> >top folder.  ./harness -help will print a help message, and the README
> >file gives more detailed instructions, but it is very intuitive.  Simply
> >specify the tests or folders you want to run.
> >
> >"./harness javax.swing" will run all the tests in
> >gnu.testlet.javax.swing.  
> >
> >It is important to specify the VM you wish to use to run the tests.  If
> >you always (or almost always) run the Mauve tests using the same VM, you
> >should export the environment variable MAUVEVM.  For example, I have
> >MAUVEVM=jamvm, so the above command would run all the javax.swing tests
> >on JamVM.  Alternately, you can use the -vm option (say, if you want to
> >run against Sun to make sure the tests pass, or just for comparison):
> >
> >"./harness -vm PATH-TO-EXECUTABLE" javax.swing"
> >
> >This will override the MAUVEVM variable.  If neither of these options is
> >used, the tests will be run on the system "java" VM, whatever you have
> >that set to.
> >
> >Comments are greatly appreciated.
> >
> >  
> >
> Hi Anthony,
> 
> I gave this a quick run - it's very nice!  I really like how easy you've 
> made it to run all the tests for one class or package. 
> 
> I have a few comments:
> 
> - is there an easy way to run all the tests?  Maybe something like 
> "./harness gnu.testlet -vm jamvm"?  Perhaps I missed something obvious, 
> but if this is possible then you could add it to the README.
> 
> - compiling the code manually was straightforward, but perhaps it could 
> be done in one step with "make" or "ant", just to make it as simple as 
> possible?

Agreed.  I think we should have a configure script and Makefile because
people expect those.  I envision:

./configure --with-java=jamvm --with-javac=ecj
make
make check

--with-java would specify the default VM to test and --with-javac would
specify the build compiler.

make would build the harness

make check would run all the tests against the default VM (in other
words, it would just run ./harness).

Note that Anthony is going to have the harness compile tests, in which
case make would only need to build the harness itself.

> 
> - this runner ignores the tags in the tests (JDK1.0, JDK1.1, GUI etc.) 
> unless you use the -file option.  Do you have any plans to add support 
> for the flags?  I don't know how much it matters, as I get the feeling 
> that the tags are not used a lot.

Yes, I think we should deprecate tags support.  From what I hear tags
were useful when testing embedded VMs on space-limited target boards and
when use of reflection was disallowed.  But reflection obsoletes the
not-a-test tag, and modern compilers obsolete the Uses: lines.  The GUI
tag can be eliminated too since we can now -exclude java.awt and
-exclude javax.swing.  Likewise GNU-CRYPTO tagged tests can be specified
by package instead.

I don't know if we should actually remove the obsoleted tags, since
they're harmless (thoughts?).  I do think the we should keep and
maintain the JDK version tags since they're useful in the same way that
@since javadoc fields are.

I would like to see the scripts and harnesses that use tags removed from
HEAD since they'll only confuse new users -- everyone should use the
same harness and we should focus all effort on improving and extending
it.  To proceed with that we must first fulfill libgcj's test harness
requirements, which still relies on tag support.

Tom