Re: [PATCH] Add compile testing to glibc test framework.

[Carlos O'Donell <carlos at redhat dot com>]

On 06/07/2016 01:28 PM, Joseph Myers wrote:
> On Tue, 7 Jun 2016, Carlos O'Donell wrote:
> 
>> Previously these kinds of tests would result in the testsuite failing
>> to run to completion because compiler errors are treated as harness
>> failures.

I agree that tests-compile is quite hackish and I'm happy to make all
of these tests just like normal tests.

> I don't actually see this as a problem - that is, I don't see why any 
> compile failure should be hard to fix for some system-specific reason.  
> I'd rather just add such tests as normal tests, that break the build if 
> they fail.

This would require that users use the latest kernel headers for all of
their builds or `make check` would fail. I always use the latest kernel
headers but I don't want to force that for all developers, and it would
be a change in the way the average developer works.

How about this?

* Extend evaluate-test.sh to support return code 88 as a dynamic xfail
  - Played around with 99 for tests that must not be xfailed.
* Extend kernel-features.h to define __LINUX_KERNEL_HEADER_VERSION the
  version of the headers used in the build.
  - Define __TEST_* macros for features that should be tested based on
    __LINUX_KERNEL_HEADER_VERSION.
* Write tests that that use __TEST_* macros and include linux and glibc
  headers. This way if you have a new enough set of kernel headers the
  tests will attempt to verify the header compatibility. Otherwise if
  your headers are not new enough you see XFAILd tests for these.
* As new compat is added we add more tests and newer __TEST_* macros
  enabled for newer kernel headers.

Example patch below.

Thoughts?

-- 
Cheers,
Carlos.

diff --git a/scripts/evaluate-test.sh b/scripts/evaluate-test.sh
index f2c85ed..b206b53 100755
--- a/scripts/evaluate-test.sh
+++ b/scripts/evaluate-test.sh
@@ -24,21 +24,35 @@ rc=$2
 orig_rc=$rc
 xfail=$3
 stop_on_failure=$4
+xfailable=false
 
-if [ $rc -eq 77 ]; then
-  result="UNSUPPORTED"
-  rc=0
-else
-  if [ $rc -eq 0 ]; then
-    result="PASS"
-  else
+case $rc in
+  99)
+    # Hard fail. Cannot be xfailed.
     result="FAIL"
-  fi
-
-  if $xfail; then
-    result="X$result"
+    ;;
+  88)
+    # Dynamic XFAIL.
+    result="XFAIL"
     rc=0
-  fi
+    ;;
+  77)
+    # Skip.
+    result="UNSUPPORTED"
+    rc=0
+    ;;
+  0)
+    result="PASS"
+    xfailable=true
+    ;;
+  *)
+    result="FAIL"
+    xfailable=true
+    ;;
+esac
+if $xfail && $xfailable; then
+  result="X$result"
+  rc=0
 fi
 
 echo "$result: $test_name"
diff --git a/sysdeps/unix/sysv/linux/kernel-features.h b/sysdeps/unix/sysv/linux/kernel-features.h
index 02c530b..7373001 100644
--- a/sysdeps/unix/sysv/linux/kernel-features.h
+++ b/sysdeps/unix/sysv/linux/kernel-features.h
@@ -17,6 +17,25 @@
    License along with the GNU C Library; if not, see
    <http://www.gnu.org/licenses/>.  */
 
+/* There are two important constants used in this file:
+   * __LINUX_KERNEL_HEADER_VERSION
+     - Version of the linux kernel headers used to compile glibc.
+     - Used to determine when a test can assume a feature is present
+       for testing purposes.
+   * __LINUX_KERNEL_VERSION
+     - Minimum version of the linux kernel usable by glibc.
+     - Used to determine when code can assume a feature is present
+       at runtime.  */
+
+/* Provide LINUX_VERSION_CODE.  */
+#include <linux/version.h>
+
+#ifndef LINUX_VERSION_CODE
+# error "Linux kernel does not define a usable version."
+#else
+# define __LINUX_KERNEL_HEADER_VERSION LINUX_VERSION_CODE
+#endif
+
 /* This file must not contain any C code.  At least it must be protected
    to allow using the file also in assembler files.  */
 
@@ -151,3 +170,17 @@
    separate syscalls were only added later.  */
 #define __ASSUME_SENDMSG_SYSCALL	1
 #define __ASSUME_RECVMSG_SYSCALL	1
+
+/* On Linux kernel 3.12 or newer the IPv6 headers are guarded to
+   ensure compatibiltiy with glibc INET headers. You need at least
+   Linux kernel 3.19 to get the full set of compatible headers for
+   glibc inet/in.h.  */
+#if __LINUX_KERNEL_HEADER_VERSION >= 0x031300
+#define __TEST_USE_KERNEL_IPV6_DEFS	1
+#else
+#define __TEST_USE_KERNEL_IPV6_DEFS	0
+#endif
+
+/* No Linux kernel has fixes for INET header testing.
+   XFAIL these tests.  */
+#define __TEST_USE_KERNEL_INET_DEFS	0
diff --git a/sysdeps/unix/sysv/linux/tst-ipv6-compat1.c b/sysdeps/unix/sysv/linux/tst-ipv6-compat1.c
new file mode 100644
index 0000000..a138982
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/tst-ipv6-compat1.c
@@ -0,0 +1,83 @@
+/* Test IPv6 linux kernel and glibc header interoperability.
+   Copyright (C) 2016 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+#include <kernel-features.h>
+
+/* Run the first test by default.  */
+#ifndef TST_IPV6_COMPAT
+# define TST_IPV6_COMPAT 1
+#endif
+
+#if __TEST_USE_KERNEL_IPV6_DEFS
+
+/* Application has to compile and run successfully.  */
+#define RET 0
+
+#if TST_IPV6_COMPAT == 1
+/* Test the basic coordinated IPv6 definitions:
+   Test netinet/in.h (glibc) coordinates with linux/in6.h (linux) (1/2).  */
+#include <netinet/in.h>
+#include <linux/in6.h>
+#endif
+
+#if TST_IPV6_COMPAT == 2
+/* Test the basic coordinated IPv6 definitions.
+   Test netinet/in.h (glibc) coordinates with linux/in6.h (linux) (2/2).  */
+#include <linux/in6.h>
+#include <netinet/in.h>
+#endif
+
+#if TST_IPV6_COMPAT == 3
+/* Test the coordinated IPv6 definitions:
+   Test netdb.h (glibc) coordinates with linux/xfrm.h (linux) (1/2).  */
+#include <netdb.h>
+#include <linux/xfrm.h>
+#endif
+
+#if TST_IPV6_COMPAT == 4
+/* Test the coordinated IPv6 definitions:
+   Test netdb.h (glibc) coordinates with linux/xfrm.h (linux) (2/2).  */
+#include <linux/xfrm.h>
+#include <netdb.h>
+#endif
+
+#if TST_IPV6_COMPAT == 5
+/* Test the coordinated IPv6 definitions:
+   Test netinet/in.h (glibc) coordinates with linux/ipv6.h (linux) (2/2).  */
+#include <netinet/in.h>
+#include <linux/ipv6.h>
+#endif
+
+#if TST_IPV6_COMPAT == 6
+/* Test the coordinated IPv6 definitions:
+   Test netdb.h (glibc) coordinates with linux/xfrm.h (linux) (2/2).  */
+#include <linux/ipv6.h>
+#include <netinet/in.h>
+#endif
+
+#else
+
+/* The kernel headers don't support coordinated IPv6 definitions.  */
+#define RET 88  /* XFAIL.  */
+
+#endif
+
+int
+main (void)
+{
+  return RET;
+}
diff --git a/sysdeps/unix/sysv/linux/tst-ipv6-compat2.c b/sysdeps/unix/sysv/linux/tst-ipv6-compat2.c
new file mode 100644
index 0000000..7bdad6f
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/tst-ipv6-compat2.c
@@ -0,0 +1,2 @@
+#define TST_IPV6_COMPAT 2
+#include "tst-ipv6-compat1.c"