[2.20] [5/6] Generate overall summary of test results

["Joseph S. Myers" <joseph at codesourcery dot com>]

The previous patches in this series having made all tests generate
.test-result files, and made an enumeration of tests available in
makefile variables, it is now straightforward to generate an overall
summary of test results, which this patch does.  A new script
merge-test-results.sh deals both with collecting results within a
directory to a file with all the results from that directory, and
collecting the results from subdirectories into a single overall file
(there's not much in common between the two modes of operation of the
script, but it seemed silly to have two separate scripts for this).
Within a directory, missing results produce UNRESOLVED lines; at top
level, missing results for a whole directory produce an ERROR line
(since toplevel can't identify what the specific missing tests are in
this case).

Note that as with previous patches in this series, this does not
change the rules for when "make" considers there has been an error, or
terminates, so unexpected failures will still cause make to terminate,
or, with -k, mean the commands for "tests" don't get run because of
failure of a dependency.  Patch 6 will change things so that test
failures do not count as errors to make and useful logs can be
generated in the presence of failures.

Questions:

* Is the use of a single merge-test-results.sh script appropriate or
  should it be two separate scripts for the two ways it is used?

* In the toplevel Makefile, the rule for xtests is more minimal than
  that for tests because there are no tests for xtests at top level.
  Is this appropriate, or should it use $(xtests-special) despite that
  variable being empty, and otherwise mimic the rule for tests?  Or,
  should we say that it's a deprecated feature to have tests at top
  level at all, that such tests would best be moved into
  subdirectories in separate patches, and so that keeping the xtests
  rule minimal is the right thing to do because it shows what the
  tests rule should look like once tests have been moved away from
  toplevel?

  (c++-types-check.out may not have anywhere better to go than misc/,
  as a miscellaneous global test.  check-local-headers.out could
  reasonably become a test run in each directory, checking that
  directory's .d files, I think, and so could begin-end-check.out,
  checking the actual headers that directory would install rather than
  a separate hardcoded list of headers - you get the issue of the
  headers installed at toplevel, but those might just as well be
  installed from a makefile in include/, and in any case they aren't
  currently covered by begin-end-check anyway.)

Tested x86_64, including that the summary does in fact reflect all the
tests with .test-result files.

2014-01-10  Joseph Myers  <joseph@codesourcery.com>

	* scripts/merge-test-results.sh: New file.
	* Makefile (tests-special-notdir): New variable.
	(tests): Run merge-test-results.sh.
	(xtests): Likewise.
	* Rules (tests-special-notdir): New variable.
	(xtests-special-notdir): Likewise.
	(tests): Run merge-test-results.sh
	(xtests): Likewise.

diff --git a/Makefile b/Makefile
index 719f5b7..8434c00 100644
--- a/Makefile
+++ b/Makefile
@@ -318,7 +318,18 @@ $(objpfx)begin-end-check.out: scripts/begin-end-check.pl
 	$(evaluate-test)
 endif
 
+tests-special-notdir = $(notdir $(tests-special))
 tests: $(tests-special)
+	$(..)scripts/merge-test-results.sh -s $(objpfx) "" \
+	  $(sort $(tests-special-notdir:.out=)) \
+	  > $(objpfx)subdir-tests.sum
+	$(..)scripts/merge-test-results.sh -t $(objpfx) subdir-tests.sum \
+	  $(sort $(subdirs) .) \
+	  > $(objpfx)tests.sum
+xtests:
+	$(..)scripts/merge-test-results.sh -t $(objpfx) subdir-xtests.sum \
+	  $(sort $(subdirs)) \
+	  > $(objpfx)xtests.sum
 
 # The realclean target is just like distclean for the parent, but we want
 # the subdirs to know the difference in case they care.
diff --git a/Rules b/Rules
index a5015c3..3dbc74e 100644
--- a/Rules
+++ b/Rules
@@ -102,6 +102,17 @@ tests: $(tests:%=$(objpfx)%.out) $(tests-special)
 xtests: tests $(xtests:%=$(objpfx)%.out) $(xtests-special)
 endif
 
+tests-special-notdir = $(notdir $(tests-special))
+xtests-special-notdir = $(notdir $(xtests-special))
+tests:
+	$(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \
+	  $(sort $(tests) $(tests-special-notdir:.out=)) \
+	  > $(objpfx)subdir-tests.sum
+xtests:
+	$(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \
+	  $(sort $(xtests) $(xtests-special-notdir:.out=)) \
+	  > $(objpfx)subdir-xtests.sum
+
 ifeq ($(build-programs),yes)
 binaries-all-notests = $(others) $(sysdep-others)
 binaries-all-tests = $(tests) $(xtests) $(test-srcs)
diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
new file mode 100755
index 0000000..5a1d534
--- /dev/null
+++ b/scripts/merge-test-results.sh
@@ -0,0 +1,61 @@
+#! /bin/sh
+# Merge test results of individual tests or subdirectories.
+# Copyright (C) 2014 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+# usage: merge-test-results.sh -s objpfx subdir test-name...
+# (subdirectory tests; empty subdir at top level), or
+#        merge-test-results.sh -t objpfx subdir-file-name subdir...
+# (top-level merge)
+
+set -e
+
+type=$1
+objpfx=$2
+shift 2
+
+case $type in
+  -s)
+    subdir=$1
+    shift
+    subdir=${subdir:+$subdir/}
+    for t in "$@"; do
+      if [ -s "$objpfx$t.test-result" ]; then
+	cat "$objpfx$t.test-result"
+      else
+	echo "UNRESOLVED: $subdir$t"
+      fi
+    done
+    ;;
+
+  -t)
+    subdir_file_name=$1
+    shift
+    for d in "$@"; do
+      if [ -f "$objpfx$d/$subdir_file_name" ]; then
+	cat "$objpfx$d/$subdir_file_name"
+      else
+	echo "ERROR: test results for $d missing"
+      fi
+    done
+    ;;
+
+  *)
+    echo "unknown type $type" >&2
+    exit 1
+    ;;
+esac

-- 
Joseph S. Myers
joseph@codesourcery.com