This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Support expected failures in .test-result files


On Thu, Feb 27, 2014 at 12:18:48AM +0000, Joseph S. Myers wrote:
> This patch, an updated version of
> <https://sourceware.org/ml/libc-alpha/2014-01/msg00195.html> now
> proposed for inclusion in glibc, makes it possible for .test-result
> files for individual tests to contain XPASS and XFAIL rather than PASS
> and FAIL in cases where failure is expected.  This replaces the
> marking of two individual tests with "-" to cause them to be expected
> at makefile level to fail; evaluate-test.sh will ensure it exits with
> status 0 for an expected failure.
> 
> Tested x86_64.

Looks good to me.

Thanks,
Siddhesh

> 
> 2014-02-27  Joseph Myers  <joseph@codesourcery.com>
> 
> 	* scripts/evaluate-test.sh: Take new argument indicating whether
> 	failure is expected.
> 	* Makeconfig (evaluate-test): Pass argument to evaluate-test.sh
> 	indicating whether failure is expected.
> 	* conform/Makefile (test-xfail-run-conformtest): New variable.
> 	($(objpfx)run-conformtest.out): Don't expect to fail at makefile
> 	level.
> 	* posix/Makefile (test-xfail-annexc): New variable.
> 	($(objpfx)annexc.out): Don't expect to fail at makefile level.
> 
> diff --git a/Makeconfig b/Makeconfig
> index 21a7d6e..3709c88 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -1083,8 +1083,11 @@ endif
>  # The name to give to a test in test results summaries.
>  test-name = $(strip $(patsubst %.out, %, $(patsubst $(common-objpfx)%, %, $@)))
>  
> -# Command to output a test status line (such as PASS: test-name).
> +# Command to output a test status line (such as PASS: test-name).  If
> +# test-xfail-$(@F:.out=) has a nonempty value, the status will be
> +# XPASS or XFAIL rather than PASS or FAIL.
>  evaluate-test = $(..)scripts/evaluate-test.sh $(test-name) $$? \
> +		  $(if $(test-xfail-$(@F:.out=)),true,false) \
>  		  > $(common-objpfx)$(test-name).test-result
>  
>  endif # Makeconfig not yet included
> diff --git a/conform/Makefile b/conform/Makefile
> index 41f0bb3..40081f3 100644
> --- a/conform/Makefile
> +++ b/conform/Makefile
> @@ -30,10 +30,11 @@ tests: $(objpfx)run-conformtest.out
>  endif
>  endif
>  
> +test-xfail-run-conformtest = yes
>  $(objpfx)run-conformtest.out: run-conformtest.sh conformtest.pl \
>  			      $(wildcard data/*.h-data) \
>  			      $(wildcard data/*/*.h-data)
> -	-$(BASH) -e $< $(objpfx) $(PERL) '$(CC)' \
> +	$(BASH) -e $< $(objpfx) $(PERL) '$(CC)' \
>  	  '-I../include $(+sysdep-includes) $(sysincludes) -I..'; \
>  	$(evaluate-test)
>  
> diff --git a/posix/Makefile b/posix/Makefile
> index 304238f..b41055a 100644
> --- a/posix/Makefile
> +++ b/posix/Makefile
> @@ -233,8 +233,9 @@ tests: $(objpfx)bug-regex2-mem $(objpfx)bug-regex14-mem \
>  xtests: $(objpfx)bug-ga2-mem
>  endif
>  
> +test-xfail-annexc = yes
>  $(objpfx)annexc.out: $(objpfx)annexc
> -	-$(dir $<)$(notdir $<) '$(CC)' \
> +	$(dir $<)$(notdir $<) '$(CC)' \
>  	  '$(patsubst %,-I../%,$(sorted-subdirs)) -I../include $(+sysdep-includes) $(sysincludes) -I..' > $@; \
>  	$(evaluate-test)
>  
> diff --git a/scripts/evaluate-test.sh b/scripts/evaluate-test.sh
> index 9cb6435..c8f5012 100755
> --- a/scripts/evaluate-test.sh
> +++ b/scripts/evaluate-test.sh
> @@ -17,10 +17,12 @@
>  # License along with the GNU C Library; if not, see
>  # <http://www.gnu.org/licenses/>.
>  
> -# usage: evaluate-test.sh test_name rc
> +# usage: evaluate-test.sh test_name rc xfail
>  
>  test_name=$1
>  rc=$2
> +orig_rc=$rc
> +xfail=$3
>  
>  if [ $rc -eq 0 ]; then
>    result="PASS"
> @@ -28,6 +30,11 @@ else
>    result="FAIL"
>  fi
>  
> +if $xfail; then
> +  result="X$result"
> +  rc=0
> +fi
> +
>  echo "$result: $test_name"
> -echo "original exit status $rc"
> +echo "original exit status $orig_rc"
>  exit $rc
> 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com

Attachment: pgp7XeAX_m36e.pgp
Description: PGP signature


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]