This is the mail archive of the
mauve-patches@sourceware.org
mailing list for the Mauve project.
new Harness with compilation capabilities
- From: Anthony Balkissoon <abalkiss at redhat dot com>
- To: mauve-patches at sources dot redhat dot com
- Date: Mon, 24 Apr 2006 16:23:10 -0400
- Subject: new Harness with compilation capabilities
This new version of the Harness has auto-compilation capabilities and
also fixes a lot of bugs and performance issues that were
present in the
last one.
Please read the README file and try the Harness out when running
your
Mauve tests.
Here is a sample configuration, build, and run, using the
auto-compilation feature, and using jamvm as the vm to be
tested:
1) CONFIGURE:
./configure --enable-auto-compile
--with-classpath-install-dir=/home/abalkiss/sources/classpath/install/share/classpath --with-vm=jamvm
2) BUILD:
make
3) RUN:
(JAVA) Harness javax.swing.JTable
This will compile and run all of the tests in javax.swing.JTable
on
jamvm. You can switch the test vm easily using the -vm option:
(JAVA) Harness javax.swing.JTable
-vm /usr/lib/jvm/java-1.5.0/jre/bin/java will run the tests on
the VM
installed in the above location. The README is quite detailed
and
explains how to do many other things using the new Harness.
Please file bug reports or email me if things aren't working as
expected. Thanks.
2006-04-24 Anthony Balkissoon <abalkiss@redhat.com>
* .cvsignore: Added .ecjOut and .ecjErr.
* configure.in: Added --with-vm, --enable-auto-compile,
--with-classpath-install-dir, --with-ecj-jar options to
support
auto-compilation.
* harness: Removed this unneeded wrapper script.
* Harness.java: Rewritten to allow auto-compilation and
faster test
processing.
* Makefile.am: Added Harness.java and RunnerProcess.java
to
harness_files, added new target 'harness' that compiles
these files
as well as gnu/testlet/config.java, and added target
'all-local' to
make these compile by default.
* Makefile.in: Regenerated.
* README: Rewritten to include the new configure
options, compilation
options, and the new invocation style.
* RunnerProcess.java: Fixed comments, formatting, and:
(NOT-A-TEST-DESCRIPTION): Made this field private.
(verbose): Made this field static.
(debug): Likewise.
(exceptions): Likewise.
(report): Likewise.
(xmlfile): Likewise.
(RunnerProcess(boolean, boolean, boolean, TestReport)):
Removed this
unnecessary constructor.
(RunnerProcess()): New constructor.
* gnu/testlet/TestHarness.java:
(getAutoCompile): New method.
(getCpInstallDir): Likewise.
(getEcjJar): Likewise.
(getTestJava): Likewise.
* gnu/testlet/config.java.in:
(cpInstallDir): New field.
(autoCompile): Likewise.
(testJava): Likewise.
(ecjJar): Likewise.
(getCpInstallDir): New method.
(getAutoCompile): Likewise.
(getEcjJar): Likewise.
(getTestJava): Likewise.
--Tony
Index: .cvsignore
===================================================================
RCS file: /cvs/mauve/mauve/.cvsignore,v
retrieving revision 1.7
diff -u -r1.7 .cvsignore
--- .cvsignore 21 Dec 2005 15:07:19 -0000 1.7
+++ .cvsignore 24 Apr 2006 19:33:04 -0000
@@ -21,3 +21,5 @@
SimpleTestHarness
mauve-classpath
mauve.out
+.ecjOut
+.ecjErr
Index: Harness.java
===================================================================
RCS file: /cvs/mauve/mauve/Harness.java,v
retrieving revision 1.1
diff -u -r1.1 Harness.java
--- Harness.java 5 Apr 2006 20:11:27 -0000 1.1
+++ Harness.java 24 Apr 2006 19:33:04 -0000
@@ -23,6 +23,8 @@
* file and what it is designed to do.
*/
+import gnu.testlet.config;
+
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
@@ -30,6 +32,11 @@
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
+import java.lang.reflect.Constructor;
+import java.lang.reflect.Method;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.StringTokenizer;
import java.util.Vector;
/**
@@ -41,73 +48,183 @@
*/
public class Harness
{
- // This is the name of the program we will run to actually run the tests.
- private static final String RUNNERPROCESSNAME = "RunnerProcess";
+ // The compile method for the embedded ecj
+ private static Method ecjMethod = null;
+
+ // The string that will be passed to the compiler containing the options
+ // and the file(s) to compile
+ private static String compileString = null;
+
+ // The options to pass to the compiler, needs to be augmented by the
+ // bootclasspath, which should be the classpath installation directory
+ private static String compileStringBase = "-proceedOnError -nowarn";
+
+ // The writers for ecj's out and err streams.
+ private static PrintWriter ecjWriterOut = null;
+ private static PrintWriter ecjWriterErr = null;
+
+ // The constructor for the embedded ecj
+ private static Constructor ecjConstructor = null;
+ // The classpath installation location, used for the compiler's bootcalsspath
+ private static String classpathInstallDir = null;
+
+ // The location of the eclipse-ecj.jar file
+ private static String ecjJarLocation = null;
+
// How long a test may run before it is considered hung
- private long timeout = 60000; // 60 seconds, can be changed via -timeout flag
+ private static long runner_timeout = 60000;
+ // The command to invoke for the VM on which we will run the tests.
+ private static String vmCommand = null;
+
// Whether or not we should recurse into directories when a folder is
// specified to be tested
- private boolean recursion = true;
+ private static boolean recursion = true;
// Whether we should run in noisy mode
- private boolean verbose = false;
+ private static boolean verbose = false;
// Whether we should display one-line summaries for passing tests
- private boolean showPasses = false;
-
+ private static boolean showPasses = false;
+
+ // Whether we should compile tests before running them
+ private static boolean compileTests = true;
+
+ // Whether we should display information for failing compilations
+ private static boolean showCompilationErrors = false;
+
// The total number of tests run
- private int total_tests = 0;
+ private static int total_tests = 0;
// The total number of failing tests (not harness.check() calls)
- private int total_test_fails = 0;
+ private static int total_test_fails = 0;
// All the tests that were specified on the command line rather than
// through standard input or an input file
- private Vector commandLineTests = null;
+ private static Vector commandLineTests = null;
+
+ // The input file (possibly) supplied by the user
+ private static String inputFile = null;
// All the tests that were explicitly excluded via the -exclude option
- private Vector excludeTests = new Vector();
-
+ private static Vector excludeTests = new Vector();
+
// A way to speak to the runner process
- private PrintWriter out = null;
+ private static PrintWriter runner_out = null;
// A way to listen to the runner process
- private BufferedReader in = null;
+ private static BufferedReader runner_in = null;
// The process that will run the tests for us
- private Process runnerProcess = null;
+ private static Process runnerProcess = null;
// A watcher to determine if runnerProcess is hung
- private TimeoutWatcher watcher = null;
-
+ private static TimeoutWatcher runner_watcher = null;
+
// A flag indicating whether or not runnerProcess is hung
- private boolean testIsHung = false;
-
+ private static boolean testIsHung = false;
+
// A lock used for synchronizing access to testIsHung
- private Object lock = new Object();
-
+ private static Object runner_lock = new Object();
+
// The arguments used when this Harness was invoked, we use this to create an
// appropriate RunnerProcess
- String[] harnessArgs = null;
-
- // The path to the executable for the VM on which the tests will be run
- String vmCommand = null;
+ private static String[] harnessArgs = null;
-
- public static void main(String[] args)
+ // A convenience String for ensuring tests all have the same name format
+ private static final String gnuTestletHeader1 = "gnu" + File.separatorChar
+ + "testlet";
+
+ // A convenience String for ensuring tests all have the same name format
+ private static final String gnuTestletHeader2 = gnuTestletHeader1
+ + File.separatorChar;
+
+ /**
+ * The main method for the Harness. Parses through the compile line
+ * options and sets up the internals, sets up the compiler options,
+ * and then runs all the tests. Finally, prints out a summary
+ * of the test run.
+ *
+ * @param args the compile line options
+ * @throws Exception
+ */
+ public static void main(String[] args) throws Exception
{
- // Create a new Harness, set it up with the args, and run
- // the appropriate tests.
+ // Create a new Harness and set it up based on args.
Harness harness = new Harness();
harness.setupHarness(args);
+
+ // Start the runner process and run all the tests.
+ initProcess(args);
+ runAllTests();
+
+ // If more than one test was run, print a summary.
+ if (total_tests > 1)
+ System.out.println("\nTEST RESULTS:\n" + total_test_fails + " of "
+ + total_tests + " tests failed.");
+ else if (total_tests == 0)
+ {
+ // If no tests were run, try to help the user out by suggesting what
+ // the problem might have been.
+ System.out.println ("No tests were run. Possible reasons " +
+ "may be listed below.");
+ if (compileTests == false)
+ {
+ System.out.println("Autocompilation is not enabled, so the " +
+ "tests need to be compiled manually. You can enable " +
+ "autocompilation via configure, see the README for more " +
+ "info.\n");
+ }
+ else if (recursion == false)
+ {
+ System.out.println ("-norecursion was specified, did you " +
+ "specify a folder that had no tests in it?\n");
+ }
+ else if (excludeTests != null && excludeTests.size() > 0)
+ {
+ System.out.println ("Some tests were excluded.\nDid you use " +
+ "-exclude and exclude all tests (or all specified " +
+ "tests)? \n");
+ }
+ else
+ {
+ System.out.println ("Did you specify a test that " +
+ "doesn't exist or a folder that contains " +
+ "no tests? \n");
+ }
+ }
+ else if (total_test_fails == 0 && !showPasses && !verbose)
+ // Finally, if a single test was run and the passing result wasn't
+ // displayed, display it for the user.
+ System.out.println ("TEST RESULT: pass");
+ harness.finalize();
+ System.exit(total_test_fails > 0 ? 1 : 0);
}
- private void setupHarness(String[] args)
+ /**
+ * Sets up the harness internals before the tests are run. Parses through
+ * the compile line options and then sets up the compiler options.
+ * @param args
+ * @throws Exception
+ */
+ private void setupHarness(String[] args) throws Exception
{
- harnessArgs = args;
- String file = null;
+ // Save the arguments, we'll pass them to the RunnerProcess so it can
+ // set up its internal properties.
+ harnessArgs = args;
+
+ // Find out from configuration whether auto-compilation is enabled or not.
+ // This can be changed via the options to Harness (-compile true or
+ // -compile false).
+ compileTests = config.autoCompile.equals("yes");
+
+ // Find out from configuration which VM we're testing. This can be changed
+ // via the options to Harness (-vm VM_TO_TEST).
+ vmCommand = config.testJava;
+
+ // Now parse all the options to Harness and set the appropriate internal
+ // properties.
for (int i = 0; i < args.length; i++)
{
if (args[i].equals("-norecursion"))
@@ -116,6 +233,18 @@
verbose = true;
else if (args[i].equals("-showpasses"))
showPasses = true;
+ else if (args[i].equals("-compile"))
+ {
+ // User wants to use an input file to specify which tests to run.
+ if (++i >= args.length)
+ throw new RuntimeException("No file path after '-file'. Exit");
+ if (args[i].equals("yes") || args[i].equals("true"))
+ compileTests = true;
+ else if (args[i].equals("no") || args[i].equals("false"))
+ compileTests = false;
+ }
+ else if (args[i].equals("-showcompilefails"))
+ showCompilationErrors = true;
else if (args[i].equals("-help") || args[i].equals("--help")
|| args[i].equals("-h"))
printHelpMessage();
@@ -124,7 +253,25 @@
// User wants to use an input file to specify which tests to run.
if (++i >= args.length)
throw new RuntimeException("No file path after '-file'. Exit");
- file = args[i];
+ inputFile = args[i];
+ }
+ else if (args[i].equalsIgnoreCase("-classpath-install-dir"))
+ {
+ // User is specifying the classpath installation folder to use
+ // as the compiler's bootclasspath.
+ if (++i >= args.length)
+ throw new RuntimeException("No file path " +
+ "after '-classpath-install-dir'. Exit");
+ classpathInstallDir = args[i];
+ }
+ else if (args[i].equalsIgnoreCase("-ecj-jar"))
+ {
+ // User is specifying the location of the eclipse-ecj.jar file
+ // to use for compilation.
+ if (++i >= args.length)
+ throw new RuntimeException("No file path " +
+ "after '-ecj-jar'. Exit");
+ ecjJarLocation = args[i];
}
else if (args[i].equals("-exclude"))
{
@@ -132,22 +279,30 @@
if (++i >= args.length)
throw new RuntimeException ("No test or directory " +
"given after '-exclude'. Exit");
- if (args[i].endsWith(".java"))
- args[i] = args[i].substring(0, args[i].length() - 5);
excludeTests.add(startingFormat(args[i]));
}
+ else if (args[i].equals("-vm"))
+ {
+ // User wants to exclude some tests from the run.
+ if (++i >= args.length)
+ throw new RuntimeException ("No VMPATH" +
+ "given after '-vm'. Exit");
+ vmCommand = args[i];
+ }
else if (args[i].equals("-timeout"))
{
// User wants to change the timeout value.
if (++i >= args.length)
throw new RuntimeException ("No timeout value given " +
"after '-timeout'. Exit");
- timeout = Long.parseLong(args[i]);
+ runner_timeout = Long.parseLong(args[i]);
}
else if (args[i].charAt(0) == '-')
{
- // One of the ignored options (ie - handled by RunnerProcess)
- // such as -verbose.
+ // One of the ignored options (handled by RunnerProcess)
+ // such as -debug. Do nothing here but don't let it fall
+ // through to the next branch which would consider it a
+ // test or folder name
}
else if (args[i] != null)
{
@@ -158,74 +313,156 @@
}
}
- // Determine the VM on which we will run the tests.
- vmCommand = System.getProperties().getProperty("java.vm.exec");
- if (vmCommand == null)
- vmCommand = "java";
-
- // Start the runner process and run all the tests.
- initProcess(vmCommand, args);
- runAllTests(file, commandLineTests);
-
- if (total_tests > 1)
- System.out.println("\nTEST RESULTS:\n" + total_test_fails + " of "
- + total_tests + " tests failed.");
- else if (total_tests == 0)
+ // If ecj-jar wasn't specified, use the configuration value.
+ if (ecjJarLocation == null)
+ ecjJarLocation = config.ecjJar;
+
+ // If auto-compilation is enabled, verify that the ecj-jar location is
+ // valid.
+ if (compileTests)
{
- // If no tests were run, try to help the user out by suggesting what
- // the problem might have been.
- if (recursion == false)
- {
- System.out.println ("No tests were run.\nDid you use -norecursion " +
- "and specify a folder that had no tests in it?\n" +
- "For example, 'jamvm -norecursion javax.swing' will not " +
- "run any tests\nbecause no tests are located directly in " +
- "the javax.swing folder.\n\nTry removing the -norecursion " +
- "option. Use the -help option for more\ninformation or " +
- "read the README file.");
- }
- else if (excludeTests != null && excludeTests.size() > 0)
- {
- System.out.println ("No tests were run.\nDid you use -exclude " +
- "and exclude all tests (or all specified tests)? \n" +
- "Use the -help option for more information or " +
- "read the README file.");
- }
+ if (ecjJarLocation == null || ecjJarLocation.equals(""))
+ compileTests = false;
else
{
- System.out.println ("No tests were run.\nDid you specify a test that " +
- "doesn't exist or a folder that contains no tests? \n" +
- "Use the -help option for more information or " +
- "read the README file.");
- }
+ File testECJ = new File(ecjJarLocation);
+ if (!testECJ.exists())
+ compileTests = false;
+ }
}
- else if (total_test_fails == 0 && !showPasses)
- System.out.println ("TEST RESULT: pass");
- finalize();
- System.exit(total_test_fails > 0 ? 1 : 0);
- }
+
+ // If auto-compilation is enabled and the ecj-jar location was fine,
+ // set up the compiler options and PrintWriters
+ if (compileTests)
+ setupCompiler();
+
+ // If vmCommand is "java" it is likely that it defaulted to this value,
+ // so it wasn't set in configure (--with-vm) and it wasn't set
+ // on the command line (-vm TESTVM), so we should print a warning.
+ if (vmCommand.equals("java"))
+ System.out.println("WARNING: running tests on 'java'. To set the " +
+ "test VM, use --with-vm when\nconfiguring " +
+ "or specify -vm when running the Harness.\n");
+ }
+
+ /**
+ * Sets up the compiler by reflection, sets up the compiler options,
+ * and the PrintWriters to get error messages from the compiler.
+ *
+ * @throws Exception
+ */
+ private void setupCompiler() throws Exception
+ {
+ String classname = "org.eclipse.jdt.internal.compiler.batch.Main";
+ Class klass = null;
+ try
+ {
+ klass = Class.forName(classname);
+ }
+ catch (ClassNotFoundException e)
+ {
+ File jar = new File(ecjJarLocation);
+ if (! jar.exists() || ! jar.canRead())
+ throw e;
+
+ ClassLoader loader = new URLClassLoader(new URL[] { jar.toURL() });
+ try
+ {
+ klass = loader.loadClass(classname);
+ }
+ catch (ClassNotFoundException f)
+ {
+ throw e;
+ }
+ }
+
+ // Set up the compiler and the PrintWriters for the compile errors.
+ ecjConstructor =
+ klass.getConstructor
+ (new Class[] { PrintWriter.class, PrintWriter.class, Boolean.TYPE});
+ ecjMethod =
+ klass.getMethod
+ ("compile", new Class[]
+ { String.class, PrintWriter.class, PrintWriter.class });
+ ecjWriterOut = new PrintWriter(".ecjOut");
+ ecjWriterErr = new PrintWriter(".ecjErr");
+
+
+ // Set up the compiler options now that we know whether or not we are
+ // compiling, and print a header to the compiler error file.
+ compileStringBase += getClasspathInstallString();
+ ecjWriterErr.println("This file lists the compiler errors.\n" +
+ "Weird things will happen if the compiler's " +
+ "bootclasspath isn't properly set.\nIf the errors" +
+ " listed here seem odd, like:\n\n" +
+ " 'The type java.lang.Object cannot be resolved." +
+ " It is indirectly\n referenced from required " +
+ ".class files,'\n\nthen try setting the " +
+ "bootclasspath by using the\n" +
+ "--with-classpath-install-dir=CPINSTALLDIR option " +
+ "in configure.");
+ ecjWriterErr.println("\nThe compiler command used was: \n " +
+ compileStringBase + "\n");
+
+ }
/**
- * This method takes a String and puts it into a consistent format
- * so we can deal with all test names in the same way. It ensures
- * that tests start with "gnu.testlet" and that slashes ('/', which
- * are file separators) are replaced with dots (for use in class names).
+ * Removes the "gnu.testlet." from the start of a String.
+ * @param val the String
+ * @return the String with "gnu.testlet." removed
+ */
+ private static String stripPrefix(String val)
+ {
+ if (val.startsWith("gnu" + File.separatorChar + "testlet")
+ || val.startsWith("gnu.testlet."))
+ val = val.substring(12);
+ return val;
+ }
+
+ /**
+ * Get the bootclasspath from the configuration so it can be added
+ * to the string passed to the compiler.
+ * @return the bootclasspath for the compiler, in String format
+ */
+ private static String getClasspathInstallString()
+ {
+ String temp = classpathInstallDir;
+ if (temp == null)
+ {
+ temp = config.cpInstallDir;
+ if (temp.equals(""))
+ return temp;
+ }
+ temp = " -bootclasspath " + temp;
+ if (!temp.endsWith(File.separator))
+ temp += File.separator;
+ temp += "share/classpath";
+ return temp;
+ }
+
+ /**
+ * This method takes a String and puts it into a consistent format so we can
+ * deal with all test names in the same way. It ensures that tests start with
+ * "gnu/testlet" and that '.' are replaced by the file separator character.
* It also strips the .java or .class extensions if they are present,
* and removes single trailing dots.
- * @param val
- * @return
+ *
+ * @param val the input String
+ * @return the formatted String
*/
private static String startingFormat(String val)
{
if (val != null)
{
- val = val.replace(File.separatorChar, '.');
- if (! val.startsWith("gnu.testlet."))
- val = "gnu.testlet." + val;
- if (val.endsWith("."))
- val = val.substring(0, val.length() - 1);
if (val.endsWith(".class"))
val = val.substring(0, val.length() - 6);
+ if (val.endsWith(".java"))
+ val = val.substring(0, val.length() - 5);
+ val = val.replace('.', File.separatorChar);
+ if (val.endsWith("" + File.separatorChar))
+ val = val.substring(0, val.length() - 1);
+ if (! val.startsWith(gnuTestletHeader1))
+ val = gnuTestletHeader2 + val;
}
return val;
}
@@ -235,36 +472,60 @@
*/
static void printHelpMessage()
{
- System.out.println(
- "This is the Mauve Harness. Usage:\n\n" +
- " ./harness <options> <testcase | folder>\n" +
- " If no testcase or folder is specified, all the tests will be run. \n" +
- " It is strongly recommended that you use the -vm option or set the \n" +
- " environment variable MAUVEVM." +
- "\n\nExample: './harness -vm jamvm -showpasses javax.swing'\n" +
- " will use jamvm (located in your path) to run all the tests in the\n" +
- " gnu.testlet.javax.swing folder and will display PASSES\n" +
- " as well as FAILS.\n\nOptions:\n" +
- " -vm [vmpath]: specify the vm on which to run the tests\n" +
- " It is strongly recommended that you use this option\n" +
- " or set the MAUVEVM environment variable. See the \n" +
- " README file for more details.\n" +
- " -exclude [test|folder]: specifies a test or a folder to exclude\n" +
- " from the run\n" +
- " -norecursion: if a folder is specified to be run, don't run\n" +
- " the tests in its subfolders\n" +
- " -showpasses: display passing tests as well as failing ones\n" +
- " -exceptions: print stack traces for uncaught exceptions\n" +
- " -timeout [millis]: specifies a timeout value for the tests\n" +
- " (default is 60000 milliseconds)\n" +
- " -verbose: run in noisy mode, displaying extra information\n" +
- " -file [filename]: specifies a file that contains the names of\n" +
- " tests to be run (one per line)\n" +
- " -debug: displays some extra information for failing tests that\n" +
- " use the harness.check(Object, Object) method\n" +
- " -xmlout [filename]: specifies a file to use for xml output\n" +
- " -help: display this help message\n");
-
+ String align = "\n ";
+ String message =
+ "This is the Mauve Harness. Usage:\n\n" +
+
+ " JAVA Harness <options> <testcase | folder>\n" +
+ " If no testcase or folder is given, all the tests will be run. \n" +
+
+ // Example invocation.
+ "\nExample: 'jamvm Harness -vm jamvm -showpasses javax.swing'\n" +
+ " will use jamvm (located in your path) to run all the tests in the\n" +
+ " gnu.testlet.javax.swing folder and will display PASSES\n" +
+ " as well as FAILS.\n\nOPTIONS:\n\n" +
+
+ // Test Run Options.
+ "Test Run Options:\n" +
+ " -vm [vmpath]: specify the vm on which to run the tests." +
+ "It is strongly recommended" + align + "that you use this option or " +
+ "use the --with-test-java option when running" + align + "configure. " +
+ "See the README file for more details.\n" +
+ " -compile [yes|no]: specify whether or not to compile the " +
+ "tests before running them. This" + align + "overrides the configure" +
+ "option --disable-auto-compile but requires an ecj jar" + align +
+ "to be in /usr/share/java/eclipse-ecj.jar or specified via the " +
+ "--with-ecj-jar" + align + "option to configure. See the README" +
+ " file for more details.\n" +
+ " -timeout [millis]: specifies a timeout value for the tests " +
+ "(default is 60000 milliseconds)" +
+
+ // Testcase Selection Options.
+ "\n\nTestcase Selection Options:\n" +
+ " -exclude [test|folder]: specifies a test or a folder to exclude " +
+ "from the run\n" +
+ " -norecursion: if a folder is specified to be run, don't " +
+ "run the tests in its subfolders\n" +
+ " -file [filename]: specifies a file that contains the names " +
+ "of tests to be run (one per line)" +
+
+ // Output Options.
+ "\n\nOutput Options:\n" +
+ " -showpasses: display passing tests as well as failing " +
+ "ones\n" +
+ " -showcompilefails: display errors from the compiler when " +
+ "tests fail to compile\n" +
+ " -exceptions: print stack traces for uncaught " +
+ "exceptions\n" +
+ " -verbose: run in noisy mode, displaying extra " +
+ "information\n" +
+ " -debug: displays some extra information for " +
+ "failing tests that " +
+ "use the" + align + "harness.check(Object, Object) method\n" +
+ " -xmlout [filename]: specifies a file to use for xml output\n" +
+ "\nOther Options:\n" +
+ " -help: display this help message\n";
+ System.out.println(message);
System.exit(0);
}
@@ -273,8 +534,8 @@
//Clean up
try
{
- in.close();
- out.close();
+ runner_in.close();
+ runner_out.close();
runnerProcess.destroy();
}
catch (IOException e)
@@ -288,42 +549,41 @@
* This method sets up our runner process - the process that actually
* runs the tests. This needs to be done once initially and also
* every time a test hangs.
- * @param runtime the VM to use (ie "java", "jamvm", etc).
- * @param name the name
- * @param args
+ * @param args the compile line options for Harness
*/
- private void initProcess(String runtime, String[] args)
- {
- String command = runtime + " " + RUNNERPROCESSNAME;
-
+ private static void initProcess(String[] args)
+ {
+ StringBuilder sb = new StringBuilder(" RunnerProcess");
for (int i = 0; i < args.length; i++)
- command += " " + args[i];
-
+ sb.append(" " + args[i]);
+ sb.insert(0, vmCommand);
+
try
{
- runnerProcess = Runtime.getRuntime().exec(command);
- out = new PrintWriter(runnerProcess.getOutputStream(), true);
- in = new BufferedReader(new InputStreamReader(runnerProcess.getInputStream()));
- watcher = new TimeoutWatcher(timeout);
+ // Exec the process and set up in/out communications with it.
+ runnerProcess = Runtime.getRuntime().exec(sb.toString());
+ runner_out = new PrintWriter(runnerProcess.getOutputStream(), true);
+ runner_in =
+ new BufferedReader
+ (new InputStreamReader(runnerProcess.getInputStream()));
+
+ // Create a timer to watch this new process.
+ runner_watcher = new TimeoutWatcher(runner_timeout);
}
catch (IOException e)
{
System.err.println("Problems invoking RunnerProcess: " + e);
- finalize();
System.exit(1);
}
}
-
+
/**
* This method runs all the tests, both from the command line and from
* standard input. This is so the legacy method of running tests by
* echoing the classname and piping it to the Harness works, but so does
* a more natural "jamvm Harness <TESTNAME>".
- * @param file the file input of testnames to run
- * @param commandLineTests the Vector of tests that were specified on the
- * command line
*/
- private void runAllTests(String file, Vector commandLineTests)
+ private static void runAllTests()
{
// Run the commandLine tests. These were assembled into
// <code>commandLineTests</code> in the setupHarness method.
@@ -343,16 +603,17 @@
// Now run the standard input tests. First we determine if the input is
// coming from a file (if the -file option was used) or from stdin.
BufferedReader r = null;
- if (file != null)
+ if (inputFile != null)
// The -file option was used, so set up our BufferedReader to use the
// input file.
try
{
- r = new BufferedReader(new FileReader(file));
+ r = new BufferedReader(new FileReader(inputFile));
}
catch (FileNotFoundException x)
{
- throw new RuntimeException("Cannot find \"" + file + "\". Exit");
+ throw new
+ RuntimeException("Cannot find \"" + inputFile + "\". Exit");
}
else
{
@@ -365,7 +626,7 @@
// If no tests were specified to be run, we will run all the
// tests (except those explicitly excluded).
if (commandLineTests == null || commandLineTests.size() == 0)
- processTest("gnu.testlet.all");
+ processTest("gnu/testlet");
return;
}
}
@@ -398,19 +659,20 @@
* PASS and adds to the report, if the appropriate options are enabled.
* @param testName the name of the test
*/
- private void runTest(String testName)
+ private static void runTest(String testName)
{
+ String tn = stripPrefix(testName.replace(File.separatorChar, '.'));
String outputFromTest;
int temp = -1;
// Start the timeout watcher
- if (watcher.isAlive())
- watcher.reset();
+ if (runner_watcher.isAlive())
+ runner_watcher.reset();
else
- watcher.start();
+ runner_watcher.start();
// Tell the RunnerProcess to run test with name testName
- out.println(testName);
+ runner_out.println(testName);
while (true)
{
@@ -420,19 +682,29 @@
// thread has declared the test hung and if so ends the process.
if (testIsHung)
{
- synchronized (lock)
+ synchronized (runner_lock)
{
testIsHung = false;
}
- finalize();
- initProcess(vmCommand, harnessArgs);
+ try
+ {
+ runner_in.close();
+ runner_out.close();
+ runnerProcess.destroy();
+ }
+ catch (IOException e)
+ {
+ System.err.println("Could not close the interprocess pipes.");
+ System.exit(- 1);
+ }
+ initProcess(harnessArgs);
break;
}
try
{
- if (in.ready())
+ if (runner_in.ready())
{
- outputFromTest = in.readLine();
+ outputFromTest = runner_in.readLine();
if (outputFromTest.startsWith("RunnerProcess:"))
{
// This means the test finished properly, now have to see if
@@ -463,14 +735,14 @@
}
}
if (temp == -1)
- {
+ {
// This means the watcher thread had to stop the process
// from running. So this is a fail.
if (verbose)
- System.out.println(" FAIL: timed out. \nTEST FAILED: timeout "
- + stripPrefix(testName));
+ System.out.println(" FAIL: timed out. \nTEST FAILED: timeout " +
+ tn);
else
- System.out.println("FAIL: " + stripPrefix(testName)
+ System.out.println("FAIL: " + tn
+ "\n Test timed out. Use -timeout [millis] " +
"option to change the timeout value.");
@@ -482,7 +754,29 @@
// If the test passed and the user wants to know about passes, tell them.
if (showPasses && temp == 0 && !verbose)
- System.out.println ("PASS: "+stripPrefix(testName));
+ System.out.println ("PASS: "+tn);
+ }
+
+ /**
+ * This method handles the input, whether it is a single test or a folder
+ * and calls runTest on the appropriate .class files. Will also compile
+ * tests that haven't been compiled or that have been changed since last
+ * being compiled.
+ * @param cname the input file name - may be a directory
+ */
+ private static void processTest(String cname)
+ {
+ if (cname.equals("CVS") || cname.endsWith(File.separatorChar + "CVS")
+ || excludeTests.contains(cname)
+ || (cname.lastIndexOf("$") > cname.lastIndexOf(File.separator)))
+ return;
+
+ // If processSingleTest returns -1 then this test was explicitly
+ // excluded with the -exclude option, and if it returns 0 then
+ // the test was successfully run and we should stop here. Only
+ // if it returns 1 should we try to process cname as a directory.
+ if (processSingleTest(cname) == 1)
+ processFolder(cname);
}
/**
@@ -492,127 +786,353 @@
* multiple times.
*
* @param cname the name of the test to run
- * @param runAnyway true if we should run the test even if it doesn't end
- * with ".java"
* @return -1 if the test was explicitly excluded via the -exclude option,
* 0 if cname represents a single test, 1 if cname does not represent a
* single test
*/
- private int processSingleTest(String cname, boolean runAnyway)
+ private static int processSingleTest(String cname)
{
- if (cname.endsWith(".java"))
- {
- runAnyway = true;
- // FIXME: we need to invoke the compiler here in case the .class
- // is not present or is out of date.
-
- cname = cname.substring(0, cname.length() - 5);
- String temp = cname.replace('.', File.separatorChar) + ".class";
- File f = new File(temp);
- if (!f.exists())
- return -1;
- }
- // Avoid running tests multiple times by quitting if this method was
- // called from processDirectory and the file wasn't a .java file.
- if (!runAnyway)
- return -1;
-
// If the test should be excluded return -1, this is a signal
- // to processTest that it should quit.
+ // to processTest that it should quit.
if (excludeTests.contains(cname))
return -1;
- // Check if cname represents a single test, and if so run it.
- try
+ // If it's not a single test, return 1, processTest will then try
+ // to process it as a directory.
+ File jf = new File(cname + ".java");
+ if (!jf.exists())
+ return 1;
+
+ if (!compileTests)
{
- Class.forName(cname);
+ File cf = new File(cname + ".class");
+ if (!cf.exists())
+ {
+ // There is an uncompiled test, but the -nocompile option was given
+ // so we just skip it
+ return -1;
+ }
}
- catch (Throwable t)
+ else
{
- // This means it wasn't a single test.
- return 1;
+ // If compilation of the test fails, don't try to run it.
+ if (compileTest(cname + ".java") != 0)
+ return -1;
}
- // If we've reached here, we've found a legitimate test, so run it and then
- // return 0 to say that everything was fine.
+
runTest(cname);
return 0;
}
/**
- * This method processes all the tests in a directory. To avoid running
- * tests twice, we pass <code>false</code> as the 2nd argument to
- * processSingleTest. This method calls itself if it finds a directory.
- * @param cname the name of the directory
- */
- private void processDirectory(String cname)
- {
- cname = cname.replace('.', File.separatorChar);
- File dir = new File(cname);
- if (! dir.exists())
- return;
+ * This method processes all the tests in a folder. It does so in
+ * 3 steps: 1) compile a list of all the .java files in the folder,
+ * 2) compile those files unless compileTests is false,
+ * 3) run those tests.
+ * @param folderName
+ */
+ private static void processFolder(String folderName)
+ {
+ File dir = new File(folderName);
+ String dirPath = dir.getPath();
+ File[] files = dir.listFiles();
+ StringBuilder sb = new StringBuilder();
+ String fullPath = null;
+ boolean compilepassed = true;
- String[] filenames = dir.list();
- if (filenames == null)
+ // If files is null, it is likely that the user input an incorrect
+ // Harness command (like -test-vm TESTVM instead of -vm TESTVM).
+ if (files == null)
return;
- // Look through all the files and folders in dir, call this method
- // on any folders and call processSingleTest on any files. Since we pass
- // false as our 2nd argument to processSingleTest, we will not run tests
- // twice (ie, one for the ".java" file and one for the ".class file).
- for (int k = 0; k < filenames.length; k++)
- {
- String temp = dir.getPath() + File.separatorChar + filenames[k];
- File f = new File(temp);
- // If it's a directory, call this method on it.
- if (f.isDirectory() && recursion
- && ! excludeTests.contains(startingFormat(temp)))
- processDirectory(temp);
+ // First, compile the list of .java files.
+ int count = 0;
+ for (int i = 0; i < files.length; i++)
+ {
+ // Ignore the CVS folders.
+ String name = files[i].getName();
+ fullPath = dirPath + File.separatorChar + name;
+ if (name.equals("CVS") || excludeTests.contains(fullPath))
+ continue;
+
+ if (name.endsWith(".java") &&
+ !excludeTests.contains(fullPath.
+ substring(0, fullPath.length() - 5)))
+ {
+ count ++;
+ sb.append(' ' + fullPath);
+ }
else
- processSingleTest(temp.replace(File.separatorChar, '.'), false);
+ {
+ // Check if it's a folder, if so, call this method on it.
+ if (files[i].isDirectory() && recursion
+ && ! excludeTests.contains(fullPath))
+ processFolder(fullPath);
+ }
+ }
+
+ // Exit if there were no .java files in this folder.
+ if (count == 0)
+ return;
+
+ // Ignore the .java files in top level gnu/teslet folder.
+ if (dirPath.equals("gnu" + File.separatorChar + "testlet"))
+ return;
+
+ // Now compile all those tests in a batch compilation, unless the
+ // -nocompile option was used.
+ if (compileTests)
+ compilepassed = compileFolder(sb, folderName);
+
+ // And now run those tests.
+ runFolder(sb, compilepassed);
+ }
+
+ private static boolean compileFolder(StringBuilder sb, String folderName)
+ {
+ int result = - 1;
+ int compileFailsInFolder = 0;
+ compileString = compileStringBase + sb.toString();
+ try
+ {
+ result = compile();
+ }
+ catch (Exception e)
+ {
+ System.err.println("compilation exception");
+ e.printStackTrace();
+ result = - 1;
+ }
+
+ if (result != 0)
+ {
+ // The compilation was not successful. Since the -nowarn option was
+ // used to compile the tests and output was sent to the ".ecjErr"
+ // file, we can parse that file, exclude the tests that did not
+ // compile properly, and print out the errors to the user if they
+ // asked to see them.
+ try
+ {
+ BufferedReader errReader =
+ new BufferedReader(new FileReader(".ecjErr"));
+ String lastFailingTest = null;
+ String temp;
+ int loc;
+
+ // Go to the part in the file that relates to this folder
+ // specifically.
+ temp = errReader.readLine();
+ int len = folderName.length();
+ int index;
+ while (temp != null)
+ {
+ index = temp.indexOf(folderName);
+ if (index != -1 &&
+ (temp.lastIndexOf((int)File.separatorChar) == len + index))
+ break;
+ temp = errReader.readLine();
+ }
+
+ // If temp is null, we didn't find it. Otherwise, look for each
+ // individual failing compilation, count it as a fail, exclude it
+ // from the test run, and if -showcompilefails was used, print
+ // out the info.
+ while (temp != null)
+ {
+ // If we've reached a part of the file that pertains to another
+ // folder then break out of the loop.
+ if (temp.indexOf("gnu" + File.separatorChar + "testlet") != - 1
+ && temp.indexOf(folderName) == - 1)
+ break;
+
+
+ // Look for test names for failing tests, so we can exclude
+ // them from the run.
+ loc = temp.indexOf("gnu" + File.separatorChar + "testlet");
+ if (loc != - 1)
+ {
+ String name = temp.substring(loc);
+ String shortName = stripPrefix(name);
+ if (verbose && lastFailingTest != null)
+ System.out.println
+ ("TEST FAILED: compilation failed " + lastFailingTest);
+ lastFailingTest = shortName;
+
+ if (verbose)
+ System.out.println
+ ("TEST: " + shortName + "\n FAIL: compilation failed.");
+ else
+ System.out.println("FAIL: " + stripPrefix(name)
+ + ": compilation failed");
+ if (!showCompilationErrors)
+ System.out.println
+ (" Read .ecjErr for details or run with " +
+ "-showcompilefails");
+
+ // When a test fails to compile, we count it as failing
+ // but we do not run it.
+ compileFailsInFolder++;
+ total_test_fails++;
+ total_tests++;
+ excludeTests.add(name);
+ }
+
+ // If -showcompilefails was used, print out the info. Do not
+ // print the compiler summer (e.g. '3 problems (3 errors)').
+ if (showCompilationErrors &&
+ temp.indexOf(problemsString(compileFailsInFolder)) == -1)
+ System.out.println(" " + temp);
+
+ // Read the next line in the file.
+ temp = errReader.readLine();
+ }
+ if (verbose && lastFailingTest != null)
+ System.out.println
+ ("TEST FAILED: compilation failed " + lastFailingTest);
+ }
+ catch (FileNotFoundException fnfe)
+ {
+ }
+ catch (IOException ioe)
+ {
+ }
}
+ return result == 0;
}
+
/**
- * This method handles the input, whether it is a single test or a folder
- * and calls runTest on the appropriate .class files. Will also compile
- * tests that haven't been compiled or that have been changed since last
- * being compiled.
- * @param cname the input file name - may be a directory
+ * This method returns a String that the compiler prints as a summary
+ * after a batch compile (e.g. '3 problems (3 errors)'). This is so
+ * we can ignore it when printing compiler errors to the screen.
+ * @param fails the number of fails in the batch compilation
+ * @return the summary String
*/
- private void processTest(String cname)
+ private static String problemsString(int fails)
{
- if (cname.equals("CVS") || cname.endsWith(File.separatorChar + "CVS")
- || cname.indexOf("$") != - 1 || excludeTests.contains(cname))
- return;
-
- if (cname.equals("gnu.testlet.all"))
- cname = "gnu.testlet";
-
- // If processSingleTest returns -1 then this test was explicitly
- // excluded with the -exclude option, and if it returns 0 then
- // the test was successfully run and we should stop here. Only
- // if it returns 1 should we try to process cname as a directory.
- if (processSingleTest(cname, true) == 1)
- processDirectory(cname);
+ if (fails == 1)
+ return "1 problem (1 error)";
+ else
+ return fails + " problems (" + fails + " errors)";
}
/**
- * Removes the "gnu.testlet." from the start of a String.
- * @param val the String
- * @return the String with "gnu.testlet." removed
+ * Runs all the tests in a folder. If the tests were compiled by
+ * compileFolder, and the compilation failed, then we must check to
+ * see if each individual test compiled before running it.
+ * @param sb the StringBuffer holding a space delimited list of all the
+ * tests to run
+ * @param compilepassed true if the compilation step happened and all
+ * tests passed or if compilation didn't happen (because of -nocompile).
*/
- private static String stripPrefix(String val)
+ private static void runFolder(StringBuilder sb, boolean compilepassed)
{
- if (val.startsWith("gnu.testlet."))
- val = val.substring(12);
- return val;
+ StringTokenizer st = new StringTokenizer(sb.toString());
+ String nextTest = null;
+ boolean classExists;
+ while (st.hasMoreTokens())
+ {
+ nextTest = st.nextToken();
+ nextTest = nextTest.substring(0, nextTest.length() - 5);
+ classExists = (new File(nextTest + ".class")).exists();
+ if (classExists
+ && (compilepassed || ! excludeTests.contains(nextTest + ".java")))
+ runTest(nextTest);
+ }
}
+
+ /**
+ * This method invokes the embedded ECJ compiler to compile a single
+ * test, which is stored in compileArgs[2].
+ * @return the return value from the compiler
+ * @throws Exception
+ */
+ public static int compile () throws Exception
+ {
+ /*
+ * This code depends on the patch in Comment #10 in this bug
+ * report:
+ *
+ * https://bugs.eclipse.org/bugs/show_bug.cgi?id=88364
+ */
+
+ Object ecjInstance = ecjConstructor.newInstance (new Object[] {
+ new PrintWriter (System.out),
+ new PrintWriter (System.err),
+ Boolean.FALSE});
+ return ((Boolean) ecjMethod.invoke (ecjInstance, new Object[] {
+ compileString, ecjWriterOut, ecjWriterErr})).booleanValue() ? 0 : -1;
+ }
+
+ private static int compileTest(String testName)
+ {
+ int result = -1;
+ // Compile the tests before running them, and if compilation fails report
+ // it as a test failure.
+ try
+ {
+ compileString = compileStringBase + " " + testName;
+ result = compile();
+ }
+ catch (Exception e)
+ {
+ result = - 1;
+ }
+ if (result != 0)
+ {
+ String shortName = stripPrefix(testName);
+ if (verbose)
+ System.out.println
+ ("TEST: " + shortName + "\n FAIL: compilation failed.");
+ else
+ System.out.println("FAIL: " + stripPrefix(testName)
+ + ": compilation failed");
+ if (!showCompilationErrors)
+ System.out.println
+ (" Read .ecjErr for details or run with -showcompilefails");
+ else
+ {
+ try
+ {
+ BufferedReader errReader =
+ new BufferedReader(new FileReader(".ecjErr"));
+ String temp = errReader.readLine();
+ while(temp != null && temp.indexOf(testName) == -1)
+ temp = errReader.readLine();
+ System.out.println(" " + temp);
+ temp = errReader.readLine();
+ while (temp != null &&
+ temp.indexOf("gnu" + File.separatorChar + "teslet") == -1)
+ {
+ System.out.println(" " + temp);
+ temp = errReader.readLine();
+ }
+ }
+ catch (FileNotFoundException fnfe)
+ {
+ }
+ catch (IOException ioe)
+ {
+ }
+ }
+ if (config.cpInstallDir.equals(""))
+ System.out.println(" Try setting --with-classpath-install-dir " +
+ "when running configure.\n See the README file for details");
+ if (verbose)
+ System.out.println("TEST FAILED: compilation failed " + shortName);
+
+ total_test_fails++;
+ total_tests++;
+ result = - 1;
+ }
+ return result;
+ }
/**
* This class is used for our timer to cancel tests that have hung.
* @author Anthony Balkissoon abalkiss at redhat dot com
*
*/
- class TimeoutWatcher implements Runnable
+ private static class TimeoutWatcher implements Runnable
{
private long millisToWait;
private Thread watcherThread;
@@ -674,7 +1194,7 @@
}
// The test is hung, set testIsHung to true so the process will be
// destroyed and restarted.
- synchronized (lock)
+ synchronized (runner_lock)
{
testIsHung = true;
}
Index: Makefile.am
===================================================================
RCS file: /cvs/mauve/mauve/Makefile.am,v
retrieving revision 1.27
diff -u -r1.27 Makefile.am
--- Makefile.am 5 Nov 2004 07:52:13 -0000 1.27
+++ Makefile.am 24 Apr 2006 19:33:04 -0000
@@ -9,7 +9,9 @@
check_DATA = $(STAMP)
-harness_files = gnu/testlet/SimpleTestHarness.java \
+harness_files = Harness.java \
+ RunnerProcess.java \
+ gnu/testlet/SimpleTestHarness.java \
gnu/testlet/TestHarness.java gnu/testlet/Testlet.java \
gnu/testlet/TestSecurityManager.java \
gnu/testlet/TestSecurityManager2.java \
@@ -17,6 +19,12 @@
gnu/testlet/TestReport.java \
gnu/testlet/TestResult.java
+harness: FORCE
+ $(JAVAC) $(harness_files)
+ $(JAVAC) gnu/testlet/config.java
+
+all-local: harness
+
## FIXME: leading space makes automake ignore this. Bleah.
include choices
Index: README
===================================================================
RCS file: /cvs/mauve/mauve/README,v
retrieving revision 1.22
diff -u -r1.22 README
--- README 5 Apr 2006 21:20:07 -0000 1.22
+++ README 24 Apr 2006 19:33:04 -0000
@@ -7,67 +7,133 @@
given runtime.
-******BUILDING THE TESTSUITE*******
-To build, first run configure. You can control the configuration with
-some environment variables:
+This file has two main sections:
+0) CONFIGURING AND BUILDING THE TESTSUITE
+1) RUNNING THE TESTS
+
+
+
+******CONFIGURING AND BUILDING THE TESTSUITE*******
+To build, first run configure. Below we cover the following:
+ 0. Specifying which VM to test.
+ 1. Configuring the auto-compilation feature.
+ 2. Environment variables for configure.
+ 3. Other options to configure.
+ 4. Building the testsuite.
+
+
+0. Specifying the VM to test.
+ The configure script takes the following option that specifies which java
+ implementation the testsuite will test:
+
+ --with-vm=TESTVM
+
+ For example, specifying --with-vm=jamvm will invoke "jamvm" when
+ running the tests, and will therefore test the java implementation associated
+ with your jamvm installation. If this option is not specified, it defaults
+ to "java".
+
+
+1. Configuring the auto-compilation feature.
+ If you plan to use the auto-compilation feature of the Harness, which
+ compiles tests inline before running them (so new tests, or changed
+ tests are run properly without an additional step), you need to be sure
+ it is properly configured. The following options are associated
+ with auto-compilation:
+
+ --enable-auto-compilation Enable the auto-compilation feature
+ --with-ecj-jar Specify the location of the eclipse-ecj.jar
+ file for the compiler to use. This defaults
+ to /usr/share/java/eclipse-ecj.jar and only
+ needs to be specified if the location on your
+ machine is different from this. If no valid
+ ecj jar is found, auto-compilation will be
+ turned off.
+ --with-classpath-install-dir Specify the location of the java implemenation
+ installation. This will be used as the
+ bootclasspath for the compiler.
+
+
+2. Environment variables for configure.
+ You can control the configuration with some environment variables:
JAVA Name of Java interpreter to use
JAVAC Name of Java (to class) compiler to use
GCJ Name of Java (to object) compiler to use
-GCJ is only used when the `--with-gcj' option is given to configure.
-
-The configure script also supports the following `--with' options:
+ GCJ is only used when the `--with-gcj' option is given to configure.
+
+3. Other configure options.
+ The configure script also supports the following `--with' options:
+
+ --with-gcj Indicate you will be using gcj, this will tell
+ make to build the object files and use the
+ appropriate compiler
--with-tmpdir=DIR Put temporary files in DIR
defaults to `/tmp'
--with-mailhost=HOSTNAME Use mail server at HOSTNAME for socket tests
defaults to `mx10.gnu.org'
(Use this option if your local firewall
blocks outgoing connections on port 25.)
-
-After configuring, compile (using your favourite java compiler) the files
-Harness.java and RunnerProcess.java.
-
-Then compile all the tests (or, if you don't plan on running all the tests,
-only the ones you wish to run). This can be done by compiling *.java from
-the gnu.testlet folder. On my machine I did this with the following command:
-
-find gnu/testlet -name "*.java" | xargs jikes
+
+4. Building the testsuite.
+ If you are using the auto-compilation feature, there is no need to build the
+ tests, it will suffice to compile the files Harness.java and
+ RunnerProcess.java.
+
+ If you are using gcj or are not using the auto-compilation feature, you should
+ now run GNU make to build the testsuite.
+
+
+
*******RUNNING THE TESTS*******
-The Mauve tests are run using the harness script in the top directory. To run
-all the tests using your system default "java" VM, type "./harness".
-
-This is rarely what developers want to do, so below we provide details on how
-to do the following:
+The Mauve tests are run using the harness script in the top directory. This
+section provides instructions on how to perform the following tasks:
+ 0. Running all the tests.
1. Specify the VM on which to run the tests
2. Select a subset of the tests to run
3. Use an input file to specify the tests to run
4. Change the timeout interval
5. Change the information displayed when tests are run
+ 6. Turn on (or off) test auto-compilation.
-1. Specifying the VM on which to run the tests
+0. Running all the tests
+ To run all the tests in the Mauve test suite, type 'HARNESSVM Harness' where
+ HARNESSVM is the VM you wish to use to run the Harness. Note that this is NOT
+ the same as the VM on which you run the tests. Read item 1 to learn how to
+ set the VM on which you run the tests.
+
+ You can also run all the tests by typing 'HARNESSVM Harness gnu.testlet'.
+
- This can be done in 2 ways: setting the environment variable MAUVEVM, or
- using the -vm [vmpath] option when running the harness script. The latter
- overrides the former, so you can set MAUVEVM to be the VM that you usually
- use, and can use the -vm option when you occasionally want to use another VM.
+1. Specifying the VM on which to run the tests
+ The VM on which to run the tests can be specified by configure as described
+ in the first section, CONFIGURING AND BUILDING THE TESTSUITE. It can also be
+ done via command line options to the Harness (which override the configure
+ options). To set the VM via the command line, use:
+
+ -testvm [vmpath]
+
If, for example, I wanted to run all the JTable tests using JamVM, and then
run them all on Sun's VM for comparison, I would type:
- ./harness javax.swing.JTable -vm jamvm
+ HARNESSVM Harness javax.swing.JTable -vm jamvm
and then
- ./harness javax.swing.JTable -vm java
+ HARNESSVM Harness javax.swing.JTable -vm java
if "java" was a system command to run Sun's VM. If not, you should specify
- the path to Sun's "java" executable (ex: /usr/lib/java-1.5.0/bin/java).
+ the path to Sun's "java" executable (ex: /usr/lib/java-1.5.0/bin/java). Note
+ again that HARNESSVM is just the VM used to run the harness, and not the tests.
+ This is done for performance considerations to allow the Harness to be natively
+ compiled and ran while testing a variety of VMs.
2. Selecting a subset of the tests to run
@@ -81,10 +147,10 @@
Example: run all the java.net tests (remember, this uses system default
"java" unless you have environment variable MAUVEVM set):
- 1. ./harness java.net
- 2. ./harness gnu.testlet.java.net
- 3. ./harness gnu/testlet/java/net
- 3. ./harness gnu/testlet/java/net/
+ 1. HARNESSVM Harness java.net
+ 2. HARNESSVM Harness gnu.testlet.java.net
+ 3. HARNESSVM Harness gnu/testlet/java/net
+ 3. HARNESSVM Harness gnu/testlet/java/net/
* It makes no difference if you use "." or "/", or if you have the
"gnu.testlet" preceeding the test folder or if you have a trailing "/".
@@ -92,8 +158,8 @@
-exclude option. Extending our previous example, let's run all the java.net
tests except for the java.net.Socket tests.
- 1. ./harness java.net -exclude java.net.Socket
- 2. ./harness -exclude java.net.Socket java.net
+ 1. HARNESSVM Harness java.net -exclude java.net.Socket
+ 2. HARNESSVM Harness -exclude java.net.Socket java.net
The test or folder you want to exclude must follow the -exclude option, but
other than that, the order doesn't matter. In example #2 above java.net is
@@ -106,9 +172,9 @@
all. So to run the AbstractDocument tests but not the BranchElement or
LeafElement tests, type:
- ./harness javax.swing.text.AbstractDocument -norecursion
+ HARNESSVM Harness javax.swing.text.AbstractDocument -norecursion
- Again, the order of the arguments/options after ./harness doesn't matter.
+ Again, the order of the arguments/options doesn't matter.
3. Using an input file to specify tests to be run
@@ -116,12 +182,12 @@
Simply use the -file [filename] option. The input file should list, one per
line, the tests or folders you want to test.
- Example: ./harness -file myInputFile
+ Example: HARNESSVM Harness -file myInputFile
The input file specifies only tests to be run, not excluded, to exclude tests
you need to explicitly do so as in Section 2 above.
- Example: ./harness -file myInputFile -exclude java.net.Socket
+ Example: HARNESSVM Harness -file myInputFile -exclude java.net.Socket
4. Changing the timeout interval
@@ -134,7 +200,7 @@
you may want to decrease the interval. To set the timeout interval use
the -timeout [interval] option. The interval is specified in milliseconds.
- Example: ./harness gnu.java.security -timeout 30000
+ Example: HARNESSVM Harness gnu.java.security -timeout 30000
will set the timeout to be 30 seconds instead of 60.
@@ -154,5 +220,36 @@
-debug: prints toString() information when a
harness.check(Object, Object) call fails.
+
+
+6. Turning on (or off) test auto-compilation
+ The auto-compilation feature is diabled by default, but can be turned on by
+ using the --enable-auto-compilation option to configure. You can also turn
+ on by using the following Harness option:
+
+ -compile yes OR -compile true
+
+ If you have enabled auto-compilation in configure but wish to turn it off
+ for a particular test run, use
+
+ -compile no OR -compile false
+
+ Note that auto-compilation requires a correct path to eclipse-ecj.jar and
+ a correct bootclasspath. The following options deal with this:
+
+ -classpath-install-dir CP_INSTALL_DIR
+ -ecj-jar ECJ_JAR_LOCATION
+
+ These can also be specified at configure time via
+ --with-classpath-installation-dir and --with-ecj-jar, the latter defaulting
+ to /usr/share/java/eclipse-ecj.jar if no location is specified.
+
+ The following example is for the case where auto-compilation is disabled by
+ default in configure, but is turned on after a cvs update so that new tests
+ are compiled and run.
+ Example:
+ - configure using --with-classpath-install-dir=CP_INSTALL_DIR
+ - cvs update (downloads the new testcases)
+ - HARNESSVM Harness -compile true
Index: RunnerProcess.java
===================================================================
RCS file: /cvs/mauve/mauve/RunnerProcess.java,v
retrieving revision 1.1
diff -u -r1.1 RunnerProcess.java
--- RunnerProcess.java 5 Apr 2006 20:11:27 -0000 1.1
+++ RunnerProcess.java 24 Apr 2006 19:33:04 -0000
@@ -45,7 +45,7 @@
extends TestHarness
{
// A description of files that are not tests
- public static final String NOT_A_TEST_DESCRIPTION = "not-a-test";
+ private static final String NOT_A_TEST_DESCRIPTION = "not-a-test";
// Total number of harness.check calls since the last checkpoint
private int count = 0;
@@ -66,14 +66,15 @@
private int total = 0;
// True if we should run in verbose (noisy) mode
- private boolean verbose = false;
+ private static boolean verbose = false;
+
// True if failing calls to harness.check(Object, Object) should print the
// toString methods of each Object
- private boolean debug = false;
+ private static boolean debug = false;
// True if stack traces should be printed for uncaught exceptions
- private boolean exceptions = false;
+ private static boolean exceptions = false;
// A description of the test
private String description;
@@ -82,20 +83,17 @@
private String last_check;
// The TestReport if a report is necessary
- private TestReport report = null;
+ private static TestReport report = null;
+ // The xmlfile for the report
+ private static String xmlfile = null;
+
// The result of the current test
private TestResult currentResult = null;
-
- protected RunnerProcess(boolean verbose, boolean debug,
- boolean exceptions, TestReport report)
- {
- this.verbose = verbose;
- this.debug = debug;
- this.exceptions = exceptions;
- this.report = report;
-
+
+ protected RunnerProcess()
+ {
try
{
BufferedReader xfile = new BufferedReader(new FileReader("xfails"));
@@ -115,16 +113,10 @@
}
}
- public static void main(String[] args)
+ public static void main(String[] args) throws Exception
{
- boolean verbose = false;
- boolean debug = false;
- boolean exceptions = false;
- String xmlfile = null;
- TestReport report = null;
-
// The test that Harness wants us to run.
- String testname;
+ String testname = null;
// This reader is used to get testnames from Harness
BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
@@ -163,14 +155,21 @@
try
{
testname = in.readLine();
- RunnerProcess harness =
- new RunnerProcess(verbose, debug, exceptions, report);
- runAndReport(harness, testname, report, xmlfile);
+ RunnerProcess harness = new RunnerProcess();
+ runAndReport(harness, testname);
}
catch (IOException ioe)
{
- System.out.println("Problems communicating between " +
- "Harness and RunnerProcess");
+ String shortName = stripPrefix(testname);
+ if (verbose)
+ System.out.println ("TEST: " + shortName +
+ "\n FAIL: failed to load\n" +
+ "TEST FAILED: failed to load "+
+ shortName);
+ else
+ System.out.println("FAIL: " + stripPrefix(testname)
+ + ": failed to load");
+ System.out.println("RunnerProcess:fail");
}
}
}
@@ -194,7 +193,6 @@
try
{
Class k = Class.forName(name);
-
Object o = k.newInstance();
if (! (o instanceof Testlet))
{
@@ -249,6 +247,7 @@
++total;
}
+ // If the harness started okay, now we run the test.
if (t != null)
{
description = name;
@@ -280,30 +279,26 @@
report.addTestResult(currentResult);
}
-
/**
* This method runs a single test in a new Harness and increments the
* total tests run and total failures, if the test fails. Prints
* PASS and adds to the report, if the appropriate options are enabled.
* @param harness the TestHarness to use for this test
* @param testName the name of the test
- * @param report the TestReport to generate
- * @param xmlfile the name of the file for xml output
*/
- static void runAndReport(RunnerProcess harness, String testName,
- TestReport report, String xmlfile)
+ static void runAndReport(RunnerProcess harness, String testName)
{
// If this call to runtest hangs, Harness will terminate this process.
- harness.runtest(testName);
+ harness.runtest(testName.replace(File.separatorChar, '.'));
+
// If the test wasn't a real test, return and tell Harness so.
if (harness.description.equals(NOT_A_TEST_DESCRIPTION))
{
System.out.println("RunnerProcess:not-a-test");
return;
}
-
+ // Print out a summary.
int temp = harness.done();
-
// Print the report if necessary.
if (report != null)
{
@@ -343,8 +338,8 @@
}
}
- return (" line " + line + ": " + ((last_check == null) ? "" : last_check) + " ["
- + (count + 1) + "]");
+ return (" line " + line + ": " + ((last_check == null) ? "" : last_check)
+ + " [" + (count + 1) + "]");
}
protected int getFailures()
@@ -697,8 +692,8 @@
// sb holds all the information we wish to return.
StringBuilder sb =
- new StringBuilder(" line " + lineOrigin + ": " +
- (last_check == null ? "" : last_check) +
+ new StringBuilder(" " + (verbose ? "FAIL: " : "")+ "line " + lineOrigin
+ + ": " + (last_check == null ? "" : last_check) +
" [" + (count + 1) + "] -- uncaught exception:");
// If a full stack trace will be printed, this method returns no details.
Index: configure.in
===================================================================
RCS file: /cvs/mauve/mauve/configure.in,v
retrieving revision 1.14
diff -u -r1.14 configure.in
--- configure.in 15 Nov 2004 13:35:10 -0000 1.14
+++ configure.in 24 Apr 2006 19:33:05 -0000
@@ -8,10 +8,50 @@
dnl For EXEEXT.
AC_PROG_CC
+dnl Check for which JVM should be tested, default to "java"
+AC_ARG_WITH(vm,
+[ --with-vm=TESTJVM Run the tests with TESTJVM],
+TEST_JAVA="$with_vm", TEST_JAVA="java")
+AC_SUBST(TEST_JAVA)
+export MAUVEVM="$TEST_JAVA"
+
+
AC_ARG_WITH(gcj,
[ --with-gcj Use gcj to compile .class files])
AM_CONDITIONAL(USE_GCJ, test "$with_gcj" = yes)
+
+AC_ARG_WITH(classpath-install-dir,
+[ --with-classpath-install-dir=CPINSTALL Specify the location of Classpath's install dir],
+CP_INSTALL_DIR="$with_classpath_install_dir")
+if ! test -d "$CP_INSTALL_DIR"
+then
+CP_INSTALL_DIR=""
+fi
+AC_SUBST(CP_INSTALL_DIR)
+
+
+AC_ARG_WITH(ecj-jar,
+[ --with-ecj-jar=JARLOCATION Use the ecj jar found at JARLOCATION for auto-compilation],
+ECJ_JAR="$with_ecj_jar", ECJ_JAR=yes)
+if test "$ECJ_JAR" = "yes"
+then
+ECJ_JAR="/usr/share/java/eclipse-ecj.jar"
+fi
+
+AC_SUBST(ECJ_JAR)
+
+dnl auto-compilation is disabled by default because it requires the
+dnl --with-classpath-install-dir option to be used as well, and so
+dnl by disabling it, the standard "./configure" setup has a better
+dnl chance of producing meaningful results. If it were enabled
+dnl by default many tests would fail because the compiler wouldn't
+dnl have the correct bootclasspath
+AC_ARG_ENABLE(auto-compile,
+[ --enable-auto-compile Use ecj to compile tests on the fly],
+AUTO_COMPILE="$enable_auto_compile",AUTO_COMPILE="no")
+AC_SUBST(AUTO_COMPILE)
+
dnl See if the user prefers to build .class files with gcj, or whether
dnl javac will be used. The default is to use gcj if we're compiling
dnl with gcj.
Index: gnu/testlet/TestHarness.java
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/TestHarness.java,v
retrieving revision 1.24
diff -u -r1.24 TestHarness.java
--- gnu/testlet/TestHarness.java 21 Mar 2006 16:16:05 -0000 1.24
+++ gnu/testlet/TestHarness.java 24 Apr 2006 19:33:05 -0000
@@ -281,4 +281,23 @@
{
return mailHost;
}
+ public String getAutoCompile()
+ {
+ return autoCompile;
+ }
+
+ public String getCpInstallDir()
+ {
+ return cpInstallDir;
+ }
+
+ public String getEcjJar()
+ {
+ return ecjJar;
+ }
+
+ public String getTestJava()
+ {
+ return testJava;
+ }
}
Index: gnu/testlet/config.java.in
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/config.java.in,v
retrieving revision 1.4
diff -u -r1.4 config.java.in
--- gnu/testlet/config.java.in 12 Aug 2004 15:11:12 -0000 1.4
+++ gnu/testlet/config.java.in 24 Apr 2006 19:33:05 -0000
@@ -22,12 +22,20 @@
public interface config
{
+ public static final String cpInstallDir = "@CP_INSTALL_DIR@";
+ public static final String autoCompile = "@AUTO_COMPILE@";
+ public static final String testJava = "@TEST_JAVA@";
+ public static final String ecjJar = "@ECJ_JAR@";
public static final String srcdir = "@SRCDIR@";
public static final String tmpdir = "@TMPDIR@";
public static final String pathSeparator = "@CHECK_PATH_SEPARATOR@";
public static final String separator = "@CHECK_FILE_SEPARATOR@";
public static final String mailHost = "@MAIL_HOST@";
+ public abstract String getCpInstallDir ();
+ public abstract String getAutoCompile ();
+ public abstract String getTestJava ();
+ public abstract String getEcjJar ();
public abstract String getSourceDirectory ();
public abstract String getTempDirectory ();
public abstract String getPathSeparator ();