This is the mail archive of the
mauve-discuss@sources.redhat.com
mailing list for the Mauve project.
Re: Some more Mauve observations (test batch mode)
- From: Mark Wielaard <mark at klomp dot org>
- To: mauve-discuss at sources dot redhat dot com
- Date: 20 Jul 2002 01:29:42 +0200
- Subject: Re: Some more Mauve observations (test batch mode)
- References: <1027035419.6497.169.camel@elsschot>
Hi,
On Fri, 2002-07-19 at 01:36, Mark Wielaard wrote:
> What I am personally really missing is a test batch mode. It would be
> really nice to have a mechanism for running the compiler and VM for each
> test separately. Currently when the compiler cannot compile a test or if
> the VM crashes your complete test run stops. libgcj has something that
> can do that using dejagnu but I hope it can be done by a simple shell
> script.
I made something by adapting/splitting up the 'choose' script and I am
really happy with how it works. I have been testing it with jikes and
gcj -C as compilers and gij or kissme as runtime VMs. It also works for
compiling to native code (completely from source, not yet
source->class->native but that is not so interesting for me at the
moment). If someone has a working kjc and kaffe installation please try
it out. To use it just set a variable in the 'runner' an 'batch_run'
script and run it with ./batch_run > results_file. (Mauve should have
been correctly configured though.)
Attached you find the following files:
choose-classes:
- generates a 'test_classes' file given some keys/tags.
uses-list:
- generates a 'uses_files' file containing all
the java source files used by a given test.
runner:
- Runs a given test class and generates a FAIL output
when the test crashes the VM of times out.
batch_run:
- Uses choose-classes to generate a list of all the
tests in mauve. Tries to compile all the needed files
(using the uses-list script) and generates a FAIL
when compilation fails and finally runs the test using
the runner script.
There is also a small patch for SimpleTestHarness to add a -resultsonly
flag to make it only output the PASS or FAIL results which is used by
the runner script.
These scripts should be usable with the current Mauve installation.
The really tricky part is integrating them with the configure/build
environment. I have not yet really thought about it since the Makefile
looks like black magic to me. Hopefully someone is interested in helping
me integrate them...
Cheers,
Mark
#! /bin/sh
# Copyright (c) 1998, 1999 Cygnus Solutions
# Written by Tom Tromey <tromey@cygnus.com>
# Copyright (c) 2002 Free Software Foundation, Inc.
# Adapted by Mark Wielaard <mark@klomp.org>
# This file is part of Mauve.
# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING. If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA. */
# Choose the tests we want to run. Output generated in file 'test_classes'.
# Usage: choose-classes [-verbose] output-directory [tag] ...
# Run in the source directory.
# See README for information on tags.
verbose=
spectags=
tags=
file_specs=
if test "$1" = "-verbose"; then
verbose=yes
shift
fi
outdir="$1"
shift
test -z "$outdir" \
&& echo "Usage: choose-classes [-verbose] output-directory [tag] ..." \
&& exit -1
# Some tags imply other tags. For instance, JDK1.2 implies JDK1.1 and
# JDK1.0. This makes it easier to keep the tags up to date in the
# source -- we only have to keep track of differences (as opposed to
# editing every source file when a new tag is added). If we see an
# otherwise-unrecognized tag, and the file `mauve-TAG' exists, then we
# treat the contents of that file as a list of implied tags.
taglist=${1+"$@"}
seen_dashdash=
while test -n "$taglist"; do
set $taglist
i="$1"
shift
taglist=${1+"$@"}
case "$i" in
X--)
seen_dashdash=yes
;;
!!java.*)
echo "Invalid tag: $i" 1>&2
exit 1
;;
!java.* | java.* | !javax.* | javax.*)
file="`echo $i | tr . /`"
file_specs="$file_specs $file"
;;
JAVA2)
tags="$tags JDK1.0 JDK1.1 JDK1.2 $i"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
JDK1.2)
tags="$tags JDK1.0 JDK1.1 JAVA2 $i"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
JDK1.1)
tags="$tags JDK1.0 JLS1.1 $i"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
JDK1.0)
tags="$tags JLS1.0"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
JLS1.1)
tags="$tags JDK1.0 JDK1.1"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
JLS1.0)
tags="$tags JDK1.0"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
*)
file=
# Always prefer a tag file in the build directory.
if test -f "$outdir/mauve-$i"; then
file="$outdir/mauve-$i"
else
if test -f "mauve-$i"; then
file="mauve-$i"
fi
fi
if test -n "$file"; then
# We don't want tags from a file to be treated as specified
# on the command line. So we add a `X--' before the
# expansion. We don't use just `--' as this would confuse
# `set'.
taglist="$taglist X-- `sed -e 's/#.*$//g' $file`"
fi
tags="$tags $i"
test -z "$seen_dashdash" && spectags="$spectags $i"
;;
esac
done
# By default, use JDK1.1.
test -z "$tags" && tags="JDK1.0 JDK1.1"
test -n "$verbose" && echo "tags = $tags"
test -n "$verbose" && echo "spectags = $spectags"
test -n "$verbose" && echo "file_specs = $file_specs"
classes=/tmp/classes-$$
: > $classes
(cd gnu/testlet; find java javax -name '*.java' -print) |
while read file; do
exact_match=no
test -n "$verbose" && echo "Examining $file"
if test -n "$file_specs"; then
ok=yes
for pat in $file_specs; do
# PAT can be like `java.lang' or `!java.lang'.
# If `!FILE' matches `PAT', then we know the pattern
# starts with `!' and is an exclusion pattern.
# Otherwise, if `!FILE' matches `!PAT', then we know that
# the file should be included.
case "!$file" in
${pat}*)
ok=no
test -n "$verbose" && echo " ... excluded by $pat"
;;
!${pat}.java)
# If the tag list includes an exact match, then we
# unconditionally accept it.
ok=yes
exact_match=yes
test -n "$verbose" && echo " ... included by $pat (exact match)"
;;
!${pat}*)
ok=yes
test -n "$verbose" && echo " ... included by $pat"
;;
esac
done
else
ok=yes
fi
if test "$ok" = no; then
continue
fi
# Surround value with spaces so that case statements will work
# correctly.
taglist=" `grep '^// Tags:' gnu/testlet/$file | sed -e 's,^// Tags:,,'` "
istest=yes
case "$taglist" in
*\ not-a-test\ *)
istest=no
;;
esac
# For an exact match, we don't look at the tags.
if test "$exact_match" = no; then
ok=no
# If any tag on the tag list matches, then we are ok.
for tag in $tags; do
case "$taglist" in
*" ${tag} "*)
ok=yes
break
;;
esac
done
# If any specified tag hits a `not' tag, then we are not ok.
if test "$ok" = yes; then
for tag in $spectags; do
case "$taglist" in
*" !${tag} "*)
ok=no
test -n "$verbose" && echo "File $file has tag !$tag"
break
;;
esac
done
else
test -n "$verbose" && echo " ... excluded because no tag matched"
fi
fi
if test "$ok" = yes; then
class="gnu.testlet.`echo $file | sed -e 's/\.java$//' | tr / .`"
if test "$istest" = yes; then
echo $class >> $classes
test -n "$verbose" && echo "Chose $class"
else
test -n "$verbose" && echo " ... excluded because not-a-test"
fi
fi
done
if test -f $outdir/test_classes \
&& cmp $classes $outdir/test_classes > /dev/null 2>&1; then
# Files are the same.
rm $classes
else
mv $classes $outdir/test_classes
fi
exit 0
#!/bin/sh
# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>
# Based on the choose script written by Tom Tromey <tromey@cygnus.com>
# This file is part of Mauve.
# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING. If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.
# Given a (list of) test classes or java files generates a list of all java
# files used for that test in the file 'uses_files'.
usage="uses-list [-verbose] output-directory classes_or_files"
if test "$1" = "-verbose"; then
verbose=yes
shift
else
verbose=
fi
outdir="$1"
shift
test -z "$outdir" && echo "$usage" && exit -1
usesfile=/tmp/uses-file-$$
uses=/tmp/uses-$$
class="$1"
shift
test -z "$class" && echo "$usage" && exit -1
while test ! -z "$class"; do
test -n "$verbose" && echo "Processing $class"
# Strip trailing .java or .class if given and replace . with /
file="`echo $class | sed -e 's/\.java$//' | sed -e 's/\.class$//' | tr . /`"
file="$file.java"
dir="`dirname $file`"
test ! -f "$file" && echo "file $file does not exist" && exit -1
useslist=" `grep '^// Uses:' $file | sed -e 's,^// Uses:,,'` "
filelist=$file
echo "$file" >> "$usesfile"
for used in $useslist; do
usedfile=$dir/$used.java
if test ! -f "$usedfile"; then
echo "file $used file used by $file does not exist"
exit -1
fi
echo "$usedfile" >> "$usesfile"
filelist="$filelist $usedfile"
done
test -n "$verbose" && echo " ... uses: $filelist"
class="$1"
shift
done
# Now uniquify the list of auxiliary files we used.
# I think neither `uniq' nor `sort -u' is portable, so we do it by hand.
previous=
sort "$usesfile" |
while read next; do
if test "$previous" != "$next"; then
echo "$next" >> $uses
previous=$next
fi
done
if test -f $outdir/uses_files \
&& cmp $uses $outdir/uses_files > /dev/null 2>&1; then
# Files are the same.
rm $uses
else
mv $uses $outdir/uses_files
fi
rm $usesfile
#!/bin/bash
# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>
# This file is part of Mauve.
# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING. If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.
# Tries to run one test case. Produces a FAIL output if the test crashes
# or doesn't finish after some timeout occurs.
# RUNTIME command can be set in the script to select the VM.
# For testing (gcj) native code set RUNTIME to 'native' it will then try to
# execute the program SimpleTestHarness in the current directory.
# Called from batch_run script.
#
# Usage: runner test_class_name
# Run in the source directory.
#RUNTIME="gij"
RUNTIME="Kissme"
#RUNTIME="native"
# Number of seconds to wait for the test to finish
WAIT=60
# Called when timeout occurs
timeout()
{
kill -9 $PID 2>&1 > /dev/null
}
# Install timeout
trap timeout SIGALRM
sleep $WAIT && kill -s SIGALRM $$ 2>&1 > /dev/null &
TIMEOUT=$!
testclass=$1
# Run process
if test "$RUNTIME" = "native"; then
echo $testclass | ./SimpleTestHarness -resultsonly &
else
echo $testclass | $RUNTIME gnu.testlet.SimpleTestHarness -resultsonly &
fi
PID=$!
wait $PID
result=$?
# Terminated unexpectedly?
if test $result != 0 -a $result != 1; then
echo "FAIL: $testclass abnormal termination $result CRASH or TIMEOUT"
fi
# cleanup timeout
trap '' SIGALRM
kill $TIMEOUT 2>&1 > /dev/null
#!/bin/bash
# Copyright (c) 2002 Free Software Foundation, Inc.
# Written by Mark Wielaard <mark@klomp.org>
# This file is part of Mauve.
# Mauve is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2, or (at your option)
# any later version.
# Mauve is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with Mauve; see the file COPYING. If not, write to
# the Free Software Foundation, 59 Temple Place - Suite 330,
# Boston, MA 02111-1307, USA.
# Tries to compile and run all possible tests.
# Produces a FAIL output if compilation fails.
# COMPILER command can be set in the script
# For testing (gcj) native code compilation set NATIVE to 'true', otherwise
# don't set it.
# Uses runner script to run each individual test.
#
# Usage: run_batch > result_output
# Run in the source directory.
# Include every key/tag we can think of but always exclude the unicode test.
KEYS="JDK1.0 JDK1.1 JDK1.2 JDK1.3 JDK1.4 JLS1.0 JLS1.2 JDBC1.0 JDBC2.0 JAVA2"
KEYS="$KEYS !java.lang.Character.unicode"
#COMPILER="gcj -C"
#COMPILER="gcj"
COMPILER="jikes"
COMPILER_FLAGS="-g"
NATIVE=
#NATIVE="true"
# Cleanup
find gnu/testlet -name "*.class" | xargs rm -f
if test ! -z "$NATIVE"; then
rm -f SimpleTestHarness
fi
# Create Mauve Framwork
framework_sources="gnu/testlet/SimpleTestHarness.java gnu/testlet/TestHarness.java gnu/testlet/Testlet.java gnu/testlet/ResourceNotFoundException.java gnu/testlet/config.java"
if test -z "$NATIVE"; then
$COMPILER $COMPILER_FLAGS $framework_sources
fi
# Generate test classes list
./choose-classes . "$KEYS"
# Compile and run all the tests
for testclass in `cat test_classes`; do
# Get all needed java sources
./uses-list . $testclass
# Try to compile sources
if test -z "$NATIVE"; then
$COMPILER $COMPILER_FLAGS @uses_files
else
$COMPILER -o SimpleTestHarness --main=gnu.testlet.SimpleTestHarness \
$framework_sources `cat uses_files`
fi
if test $? == 0; then
./runner $testclass
else
echo "FAIL: $testclass COMPILE FAILED"
fi
done
Index: gnu/testlet/SimpleTestHarness.java
===================================================================
RCS file: /cvs/mauve/mauve/gnu/testlet/SimpleTestHarness.java,v
retrieving revision 1.29
diff -u -r1.29 SimpleTestHarness.java
--- gnu/testlet/SimpleTestHarness.java 18 Feb 2002 01:43:07 -0000 1.29
+++ gnu/testlet/SimpleTestHarness.java 19 Jul 2002 22:57:12 -0000
@@ -39,6 +39,7 @@
private int total = 0;
private boolean verbose = false;
private boolean debug = false;
+ private boolean results_only=false;
private String description;
private String last_check;
@@ -59,13 +60,13 @@
System.out.println (desc);
++failures;
}
- else if (verbose)
+ else if (verbose || results_only)
{
System.out.println ("X" + desc);
++xfailures;
}
}
- else if (verbose)
+ else if (verbose || results_only)
{
if (expected_xfails.contains (getDescription ("FAIL")))
{
@@ -235,18 +236,26 @@
protected int done ()
{
- System.out.println(failures + " of " + total + " tests failed");
- if (xpasses > 0)
- System.out.println(xpasses + " of " + total + " tests unexpectedly passed");
- if (xfailures > 0)
- System.out.println(xfailures + " of " + total + " tests expectedly failed");
+ if (!results_only)
+ {
+ System.out.println(failures + " of " + total + " tests failed");
+ if (xpasses > 0)
+ System.out.println(xpasses + " of " + total
+ + " tests unexpectedly passed");
+ if (xfailures > 0)
+ System.out.println(xfailures + " of " + total
+ + " tests expectedly failed");
+ }
return failures > 0 ? 1 : 0;
+
}
- protected SimpleTestHarness (boolean verbose, boolean debug)
+ protected SimpleTestHarness (boolean verbose, boolean debug,
+ boolean results_only)
{
this.verbose = verbose;
this.debug = debug;
+ this.results_only= results_only;
try
{
@@ -269,6 +278,7 @@
{
boolean verbose = false;
boolean debug = false;
+ boolean results_only = false;
int i;
for (i = 0; i < args.length; i++)
@@ -277,12 +287,18 @@
verbose = true;
else if (args[i].equals("-debug"))
debug = true;
+ else if (args[i].equals("-resultsonly"))
+ {
+ results_only = true;
+ verbose = false;
+ debug = false;
+ }
else
break;
}
SimpleTestHarness harness
- = new SimpleTestHarness (verbose, debug);
+ = new SimpleTestHarness (verbose, debug, results_only);
BufferedReader r
= new BufferedReader (new InputStreamReader (System.in));