Re: [2.20] [5/6] Generate overall summary of test results

["Carlos O'Donell" <carlos at redhat dot com>]

Joseph,

This is OK to checkin.

On 01/09/2014 09:14 PM, Joseph S. Myers wrote:
> The previous patches in this series having made all tests generate
> .test-result files, and made an enumeration of tests available in
> makefile variables, it is now straightforward to generate an overall
> summary of test results, which this patch does.  A new script
> merge-test-results.sh deals both with collecting results within a
> directory to a file with all the results from that directory, and
> collecting the results from subdirectories into a single overall file
> (there's not much in common between the two modes of operation of the
> script, but it seemed silly to have two separate scripts for this).
> Within a directory, missing results produce UNRESOLVED lines; at top
> level, missing results for a whole directory produce an ERROR line
> (since toplevel can't identify what the specific missing tests are in
> this case).
> 
> Note that as with previous patches in this series, this does not
> change the rules for when "make" considers there has been an error, or
> terminates, so unexpected failures will still cause make to terminate,
> or, with -k, mean the commands for "tests" don't get run because of
> failure of a dependency.  Patch 6 will change things so that test
> failures do not count as errors to make and useful logs can be
> generated in the presence of failures.
> 
> Questions:
> 
> * Is the use of a single merge-test-results.sh script appropriate or
>   should it be two separate scripts for the two ways it is used?

Please leave it as a single script for now. I agree with Brooks that
for simplicity it's nice to have it all in one script even if it's
not terribly related.

> * In the toplevel Makefile, the rule for xtests is more minimal than
>   that for tests because there are no tests for xtests at top level.
>   Is this appropriate, or should it use $(xtests-special) despite that
>   variable being empty, and otherwise mimic the rule for tests?  Or,
>   should we say that it's a deprecated feature to have tests at top
>   level at all, that such tests would best be moved into
>   subdirectories in separate patches, and so that keeping the xtests
>   rule minimal is the right thing to do because it shows what the
>   tests rule should look like once tests have been moved away from
>   toplevel?

My opinion is that there should be no top-level tests.

Any top-level tests that test features that use functionality that
is not specific to one subdirectory should instead go into their own
subdirectory that is there precisely to hold these tests.

The top-level rules, used by the entire build should be as simple
as possible.

>   (c++-types-check.out may not have anywhere better to go than misc/,
>   as a miscellaneous global test.  check-local-headers.out could
>   reasonably become a test run in each directory, checking that
>   directory's .d files, I think, and so could begin-end-check.out,
>   checking the actual headers that directory would install rather than
>   a separate hardcoded list of headers - you get the issue of the
>   headers installed at toplevel, but those might just as well be
>   installed from a makefile in include/, and in any case they aren't
>   currently covered by begin-end-check anyway.)

> Tested x86_64, including that the summary does in fact reflect all the
> tests with .test-result files.
> 
> 2014-01-10  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* scripts/merge-test-results.sh: New file.
> 	* Makefile (tests-special-notdir): New variable.
> 	(tests): Run merge-test-results.sh.
> 	(xtests): Likewise.
> 	* Rules (tests-special-notdir): New variable.
> 	(xtests-special-notdir): Likewise.
> 	(tests): Run merge-test-results.sh
> 	(xtests): Likewise.
> 
> diff --git a/Makefile b/Makefile
> index 719f5b7..8434c00 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -318,7 +318,18 @@ $(objpfx)begin-end-check.out: scripts/begin-end-check.pl
>  	$(evaluate-test)
>  endif
>  
> +tests-special-notdir = $(notdir $(tests-special))
>  tests: $(tests-special)
> +	$(..)scripts/merge-test-results.sh -s $(objpfx) "" \
> +	  $(sort $(tests-special-notdir:.out=)) \
> +	  > $(objpfx)subdir-tests.sum
> +	$(..)scripts/merge-test-results.sh -t $(objpfx) subdir-tests.sum \
> +	  $(sort $(subdirs) .) \
> +	  > $(objpfx)tests.sum
> +xtests:
> +	$(..)scripts/merge-test-results.sh -t $(objpfx) subdir-xtests.sum \
> +	  $(sort $(subdirs)) \
> +	  > $(objpfx)xtests.sum

OK.

>  
>  # The realclean target is just like distclean for the parent, but we want
>  # the subdirs to know the difference in case they care.
> diff --git a/Rules b/Rules
> index a5015c3..3dbc74e 100644
> --- a/Rules
> +++ b/Rules
> @@ -102,6 +102,17 @@ tests: $(tests:%=$(objpfx)%.out) $(tests-special)
>  xtests: tests $(xtests:%=$(objpfx)%.out) $(xtests-special)
>  endif
>  
> +tests-special-notdir = $(notdir $(tests-special))
> +xtests-special-notdir = $(notdir $(xtests-special))
> +tests:
> +	$(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \
> +	  $(sort $(tests) $(tests-special-notdir:.out=)) \
> +	  > $(objpfx)subdir-tests.sum
> +xtests:
> +	$(..)scripts/merge-test-results.sh -s $(objpfx) $(subdir) \
> +	  $(sort $(xtests) $(xtests-special-notdir:.out=)) \
> +	  > $(objpfx)subdir-xtests.sum
> +

OK.

>  ifeq ($(build-programs),yes)
>  binaries-all-notests = $(others) $(sysdep-others)
>  binaries-all-tests = $(tests) $(xtests) $(test-srcs)
> diff --git a/scripts/merge-test-results.sh b/scripts/merge-test-results.sh
> new file mode 100755
> index 0000000..5a1d534
> --- /dev/null
> +++ b/scripts/merge-test-results.sh
> @@ -0,0 +1,61 @@
> +#! /bin/sh
> +# Merge test results of individual tests or subdirectories.
> +# Copyright (C) 2014 Free Software Foundation, Inc.
> +# This file is part of the GNU C Library.
> +
> +# The GNU C Library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +
> +# The GNU C Library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with the GNU C Library; if not, see
> +# <http://www.gnu.org/licenses/>.
> +
> +# usage: merge-test-results.sh -s objpfx subdir test-name...
> +# (subdirectory tests; empty subdir at top level), or
> +#        merge-test-results.sh -t objpfx subdir-file-name subdir...
> +# (top-level merge)
> +
> +set -e

OK.

> +
> +type=$1
> +objpfx=$2
> +shift 2
> +
> +case $type in
> +  -s)
> +    subdir=$1
> +    shift
> +    subdir=${subdir:+$subdir/}
> +    for t in "$@"; do
> +      if [ -s "$objpfx$t.test-result" ]; then
> +	cat "$objpfx$t.test-result"
> +      else
> +	echo "UNRESOLVED: $subdir$t"
> +      fi
> +    done
> +    ;;
> +
> +  -t)
> +    subdir_file_name=$1
> +    shift
> +    for d in "$@"; do
> +      if [ -f "$objpfx$d/$subdir_file_name" ]; then
> +	cat "$objpfx$d/$subdir_file_name"
> +      else
> +	echo "ERROR: test results for $d missing"
> +      fi
> +    done
> +    ;;
> +
> +  *)
> +    echo "unknown type $type" >&2
> +    exit 1
> +    ;;
> +esac
> 

OK.

Cheers,
Carlos.