Re: New Harness, version 1

[David Gilbert <david dot gilbert at object-refinery dot com>]

Anthony Balkissoon wrote:

> Okay, the new Harness is checked in, please use it and comment on
> features you would like added.  
>
> Running tests is now much easier, please use the harness script in the
> top folder.  ./harness -help will print a help message, and the README
> file gives more detailed instructions, but it is very intuitive.  Simply
> specify the tests or folders you want to run.
>
> "./harness javax.swing" will run all the tests in
> gnu.testlet.javax.swing.  
>
> It is important to specify the VM you wish to use to run the tests.  If
> you always (or almost always) run the Mauve tests using the same VM, you
> should export the environment variable MAUVEVM.  For example, I have
> MAUVEVM=jamvm, so the above command would run all the javax.swing tests
> on JamVM.  Alternately, you can use the -vm option (say, if you want to
> run against Sun to make sure the tests pass, or just for comparison):
>
> "./harness -vm PATH-TO-EXECUTABLE" javax.swing"
>
> This will override the MAUVEVM variable.  If neither of these options is
> used, the tests will be run on the system "java" VM, whatever you have
> that set to.
>
> Comments are greatly appreciated.
>
>  
Hi Anthony,

I gave this a quick run - it's very nice!  I really like how easy you've 
made it to run all the tests for one class or package. 

I have a few comments:

- is there an easy way to run all the tests?  Maybe something like 
"./harness gnu.testlet -vm jamvm"?  Perhaps I missed something obvious, 
but if this is possible then you could add it to the README.

- compiling the code manually was straightforward, but perhaps it could 
be done in one step with "make" or "ant", just to make it as simple as 
possible?

- this runner ignores the tags in the tests (JDK1.0, JDK1.1, GUI etc.) 
unless you use the -file option.  Do you have any plans to add support 
for the flags?  I don't know how much it matters, as I get the feeling 
that the tags are not used a lot.

It would be nice to generate some HTML output for easy casual review, 
along the lines of:

http://www.object-refinery.com/classpath/mauve/report/

Even better than that, though, would be to get some test coverage tool 
working and have that generate some HTML reports that we can use to 
monitor our progress...

Regards,

Dave