This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH] tunables: Add IFUNC selection and cache sizes
- From: Adhemerval Zanella <adhemerval dot zanella at linaro dot org>
- To: libc-alpha at sourceware dot org
- Date: Tue, 20 Jun 2017 10:23:33 -0300
- Subject: Re: [PATCH] tunables: Add IFUNC selection and cache sizes
- Authentication-results: sourceware.org; auth=none
- References: <20170615131042.GA28885@gmail.com>
On 15/06/2017 10:10, H.J. Lu wrote:
> The current IFUNC selection is based on microbenchmarks in glibc. It
> should give the best performance for most workloads. But other choices
> may have better performance for a particular workload or on the hardware
> which wasn't available at the selection was made. The environment
> variable, GLIBC_TUNABLES=glibc.tune.ifunc=-xxx,yyy,-zzz...., can be used
> to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy and zzz,
> where the feature name is case-sensitive and has to match the ones in
> cpu-features.h. It can be used by glibc developers to override the
> IFUNC selection to tune for a new processor or improve performance for
> a particular workload. It isn't intended for normal end users.
>
> NOTE: the IFUNC selection may change over time. Please check all
> multiarch implementations when experimenting.
>
> Also, GLIBC_TUNABLES=glibc.tune.non_temporal_threshold=NUMBER is
> provided to set threshold to use non temporal store to NUMBER,
> GLIBC_TUNABLES=glibc.tune.data_cache_size=NUMBER to set data cache size,
> GLIBC_TUNABLES=glibc.tune.shared_cache_size=NUMBER to set shared cache
> size.
>
> Any comments?
>
> H.J.
> ---
> 2017-06-15 H.J. Lu <hongjiu.lu@intel.com>
> Erich Elsen <eriche@google.com>
>
> * elf/dl-tunables.list (tune): Add ifunc, non_temporal_threshold,
> data_cache_size and shared_cache_size.
> * manual/tunables.texi: Document glibc.tune.ifunc,
> glibc.tune.data_cache_size, glibc.tune.shared_cache_size and
> glibc.tune.non_temporal_threshold.
> * sysdeps/unix/sysv/linux/x86/dl-sysdep.c: New file.
> * sysdeps/x86/cpu-tunables.c: Likewise.
> * sysdeps/x86/cacheinfo.c
> (init_cacheinfo): Check and get data cache size, shared cache
> size and non temporal threshold from cpu_features.
> * sysdeps/x86/cpu-features.c [HAVE_TUNABLES] (TUNABLE_NAMESPACE):
> New.
> [HAVE_TUNABLES] Include <unistd.h>.
> [HAVE_TUNABLES] Include <elf/dl-tunables.h>.
> [HAVE_TUNABLES] (TUNABLE_CALLBACK (set_ifunc)): Likewise.
> [HAVE_TUNABLES] (init_cpu_features): Use TUNABLE_GET to set
> IFUNC selection, data cache size, shared cache size and non
> temporal threshold.
> * sysdeps/x86/cpu-features.h (cpu_features): Add data_cache_size,
> shared_cache_size and non_temporal_threshold.
> ---
> elf/dl-tunables.list | 16 ++
> manual/tunables.texi | 36 ++++
> sysdeps/unix/sysv/linux/x86/dl-sysdep.c | 21 ++
> sysdeps/x86/cacheinfo.c | 10 +-
> sysdeps/x86/cpu-features.c | 19 ++
> sysdeps/x86/cpu-features.h | 8 +
> sysdeps/x86/cpu-tunables.c | 330 ++++++++++++++++++++++++++++++++
> 7 files changed, 439 insertions(+), 1 deletion(-)
> create mode 100644 sysdeps/unix/sysv/linux/x86/dl-sysdep.c
> create mode 100644 sysdeps/x86/cpu-tunables.c
>
> diff --git a/elf/dl-tunables.list b/elf/dl-tunables.list
> index 41ce9af..78354fb 100644
> --- a/elf/dl-tunables.list
> +++ b/elf/dl-tunables.list
> @@ -82,6 +82,22 @@ glibc {
> type: UINT_64
> env_alias: LD_HWCAP_MASK
> default: HWCAP_IMPORTANT
> + }
> + ifunc {
> + type: STRING
> + security_level: SXID_IGNORE
> + }
> + non_temporal_threshold {
> + type: SIZE_T
> + security_level: SXID_IGNORE
> + }
> + data_cache_size {
> + type: SIZE_T
> + security_level: SXID_IGNORE
> + }
> + shared_cache_size {
> + type: SIZE_T
> + security_level: SXID_IGNORE
> }
Is it possible with current tunables approach to make it arch-specific? The
'ifunc' switch seems a generic one, but the 'non_temporal_threshold',
'data_cache_size', and 'shared_cache_size' are very x86 specific and I see
it confusing exposing them to non-x86 architectures.
> }
> }
> diff --git a/manual/tunables.texi b/manual/tunables.texi
> index c9a4cb7..709f58a 100644
> --- a/manual/tunables.texi
> +++ b/manual/tunables.texi
> @@ -198,6 +198,14 @@ is 8 times the number of cores online.
> @cindex hardware capability tunables
> @cindex hwcap tunables
> @cindex tunables, hwcap
> +@cindex ifunc tunables
> +@cindex tunables, ifunc
> +@cindex data_cache_size tunables
> +@cindex tunables, data_cache_size
> +@cindex shared_cache_size tunables
> +@cindex tunables, shared_cache_size
> +@cindex non_temporal_threshold tunables
> +@cindex tunables, non_temporal_threshold
>
> @deftp {Tunable namespace} glibc.tune
> Behavior of @theglibc{} can be tuned to assume specific hardware capabilities
> @@ -213,3 +221,31 @@ extensions available in the processor at runtime for some architectures. The
> @code{glibc.tune.hwcap_mask} tunable allows the user to mask out those
> capabilities at runtime, thus disabling use of those extensions.
> @end deftp
> +
> +@deftp Tunable glibc.tune.ifunc
> +The @code{glibc.tune.ifunc=-xxx,yyy,-zzz...} tunable allows the user to
> +enable CPU/ARCH feature @code{yyy}, disable CPU/ARCH feature @code{xxx}
> +and @code{zzz} where the feature name is case-sensitive and has to match
> +the ones in @code{sysdeps/x86/cpu-features.h}.
> +
> +This tunable is specific to i386 and x86-64.
> +@end deftp
> +
> +@deftp Tunable glibc.tune.data_cache_size
> +The @code{glibc.tune.data_cache_size} tunable allows the user to set
> +data cache size in bytes for use in memory and string routines.
> +
> +This tunable is specific to i386 and x86-64.
> +@end deftp
> +
> +@deftp Tunable glibc.tune.shared_cache_size
> +The @code{glibc.tune.shared_cache_size} tunable allows the user to set
> +shared cache size in bytes for use in memory and string routines.
> +@end deftp
> +
> +@deftp Tunable glibc.tune.non_temporal_threshold
> +The @code{glibc.tune.non_temporal_threshold} tunable allows the user to
> +set threshold in bytes for non temporal store.
> +
> +This tunable is specific to i386 and x86-64.
> +@end deftp
> diff --git a/sysdeps/unix/sysv/linux/x86/dl-sysdep.c b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c
> new file mode 100644
> index 0000000..64eb0d7
> --- /dev/null
> +++ b/sysdeps/unix/sysv/linux/x86/dl-sysdep.c
> @@ -0,0 +1,21 @@
> +/* Operating system support for run-time dynamic linker. X86 version.
> + Copyright (C) 2017 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#include <config.h>
> +#include <sysdeps/x86/cpu-tunables.c>
> +#include <sysdeps/unix/sysv/linux/dl-sysdep.c>
> diff --git a/sysdeps/x86/cacheinfo.c b/sysdeps/x86/cacheinfo.c
> index 8000fd1..9542dfb 100644
> --- a/sysdeps/x86/cacheinfo.c
> +++ b/sysdeps/x86/cacheinfo.c
> @@ -752,6 +752,9 @@ intel_bug_no_cache_info:
> #endif
> }
>
> + if (cpu_features->data_cache_size != 0)
> + data = cpu_features->data_cache_size;
> +
> if (data > 0)
> {
> __x86_raw_data_cache_size_half = data / 2;
> @@ -762,6 +765,9 @@ intel_bug_no_cache_info:
> __x86_data_cache_size = data;
> }
>
> + if (cpu_features->shared_cache_size != 0)
> + shared = cpu_features->shared_cache_size;
> +
> if (shared > 0)
> {
> __x86_raw_shared_cache_size_half = shared / 2;
> @@ -777,7 +783,9 @@ intel_bug_no_cache_info:
> store becomes faster on a 8-core processor. This is the 3/4 of the
> total shared cache size. */
> __x86_shared_non_temporal_threshold
> - = __x86_shared_cache_size * threads * 3 / 4;
> + = (cpu_features->non_temporal_threshold != 0
> + ? cpu_features->non_temporal_threshold
> + : __x86_shared_cache_size * threads * 3 / 4);
> }
>
> #endif
> diff --git a/sysdeps/x86/cpu-features.c b/sysdeps/x86/cpu-features.c
> index 4288001..1ca0310 100644
> --- a/sysdeps/x86/cpu-features.c
> +++ b/sysdeps/x86/cpu-features.c
> @@ -20,6 +20,15 @@
> #include <cpu-features.h>
> #include <dl-hwcap.h>
>
> +#if HAVE_TUNABLES
> +# define TUNABLE_NAMESPACE tune
> +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */
> +# include <elf/dl-tunables.h>
> +
> +extern void TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *)
> + attribute_hidden;
> +#endif
> +
> static void
> get_common_indeces (struct cpu_features *cpu_features,
> unsigned int *family, unsigned int *model,
> @@ -312,6 +321,16 @@ no_cpuid:
> cpu_features->model = model;
> cpu_features->kind = kind;
>
> +#if HAVE_TUNABLES
> + TUNABLE_GET (ifunc, tunable_val_t *, TUNABLE_CALLBACK (set_ifunc));
> + cpu_features->non_temporal_threshold
> + = TUNABLE_GET (non_temporal_threshold, long int, NULL);
> + cpu_features->data_cache_size
> + = TUNABLE_GET (data_cache_size, long int, NULL);
> + cpu_features->shared_cache_size
> + = TUNABLE_GET (shared_cache_size, long int, NULL);
> +#endif
> +
> /* Reuse dl_platform, dl_hwcap and dl_hwcap_mask for x86. */
> GLRO(dl_platform) = NULL;
> GLRO(dl_hwcap) = 0;
> diff --git a/sysdeps/x86/cpu-features.h b/sysdeps/x86/cpu-features.h
> index f2329ea..fef5e18 100644
> --- a/sysdeps/x86/cpu-features.h
> +++ b/sysdeps/x86/cpu-features.h
> @@ -215,6 +215,14 @@ struct cpu_features
> unsigned int family;
> unsigned int model;
> unsigned int feature[FEATURE_INDEX_MAX];
> + /* Data cache size for use in memory and string routines, typically
> + L1 size. */
> + long int data_cache_size;
> + /* Shared cache size for use in memory and string routines, typically
> + L2 or L3 size. */
> + long int shared_cache_size;
> + /* Threshold to use non temporal store. */
> + long int non_temporal_threshold;
> };
>
> /* Used from outside of glibc to get access to the CPU features
> diff --git a/sysdeps/x86/cpu-tunables.c b/sysdeps/x86/cpu-tunables.c
> new file mode 100644
> index 0000000..dfa3647
> --- /dev/null
> +++ b/sysdeps/x86/cpu-tunables.c
> @@ -0,0 +1,330 @@
> +/* x86 CPU feature tuning.
> + This file is part of the GNU C Library.
> + Copyright (C) 2017 Free Software Foundation, Inc.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#if HAVE_TUNABLES
> +# define TUNABLE_NAMESPACE tune
> +# include <stdbool.h>
> +# include <stdint.h>
> +# include <unistd.h> /* Get STDOUT_FILENO for _dl_printf. */
> +# include <elf/dl-tunables.h>
> +# include <string.h>
> +# include <cpu-features.h>
> +# include <ldsodefs.h>
> +
> +/* We can't use IFUNC memcmp nor strlen in init_cpu_features from libc.a
> + since IFUNC must be set up by init_cpu_features. */
> +# if defined USE_MULTIARCH && !defined SHARED
> +# ifdef __x86_64__
> +# define DEFAULT_MEMCMP __memcmp_sse2
> +# define DEFAULT_STRLEN __strlen_sse2
> +extern __typeof (strlen) DEFAULT_STRLEN;
> +# else
> +# define DEFAULT_MEMCMP __memcmp_ia32
> +# define DEFAULT_STRLEN strlen
> +# endif
> +extern __typeof (memcmp) DEFAULT_MEMCMP;
> +# else
> +# define DEFAULT_MEMCMP memcmp
> +# define DEFAULT_STRLEN strlen
> +# endif
> +
> +# define CHECK_GLIBC_IFUNC_CPU_OFF(f, cpu_features, name, len) \
> + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \
> + if (!DEFAULT_MEMCMP (f, #name, len)) \
> + { \
> + cpu_features->cpuid[index_cpu_##name].reg_##name \
> + &= ~bit_cpu_##name; \
> + break; \
> + }
> +
> +/* Disable an ARCH feature NAME. We don't enable an ARCH feature which
> + isn't available. */
> +# define CHECK_GLIBC_IFUNC_ARCH_OFF(f, cpu_features, name, len) \
> + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \
> + if (!DEFAULT_MEMCMP (f, #name, len)) \
> + { \
> + cpu_features->feature[index_arch_##name] \
> + &= ~bit_arch_##name; \
> + break; \
> + }
> +
> +/* Enable/disable an ARCH feature NAME. */
> +# define CHECK_GLIBC_IFUNC_ARCH_BOTH(f, cpu_features, name, disable, \
> + len) \
> + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \
> + if (!DEFAULT_MEMCMP (f, #name, len)) \
> + { \
> + if (disable) \
> + cpu_features->feature[index_arch_##name] \
> + &= ~bit_arch_##name; \
> + else \
> + cpu_features->feature[index_arch_##name] \
> + |= bit_arch_##name; \
> + break; \
> + }
> +
> +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only
> + if the ARCH feature NEED is also enabled. */
> +# define CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH(f, cpu_features, name, \
> + need, disable, len) \
> + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \
> + if (!DEFAULT_MEMCMP (f, #name, len)) \
> + { \
> + if (disable) \
> + cpu_features->feature[index_arch_##name] \
> + &= ~bit_arch_##name; \
> + else if (CPU_FEATURES_ARCH_P (cpu_features, need)) \
> + cpu_features->feature[index_arch_##name] \
> + |= bit_arch_##name; \
> + break; \
> + }
> +
> +/* Enable/disable an ARCH feature NAME. Enable an ARCH feature only
> + if the CPU feature NEED is also enabled. */
> +# define CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH(f, cpu_features, name, \
> + need, disable, len) \
> + _Static_assert (sizeof (#name) - 1 == len, #name " != " #len); \
> + if (!DEFAULT_MEMCMP (f, #name, len)) \
> + { \
> + if (disable) \
> + cpu_features->feature[index_arch_##name] \
> + &= ~bit_arch_##name; \
> + else if (CPU_FEATURES_CPU_P (cpu_features, need)) \
> + cpu_features->feature[index_arch_##name] \
> + |= bit_arch_##name; \
> + break; \
> + }
> +
> +attribute_hidden
> +void
> +TUNABLE_CALLBACK (set_ifunc) (tunable_val_t *valp)
> +{
> + /* The current IFUNC selection is based on microbenchmarks in glibc.
> + It should give the best performance for most workloads. But other
> + choices may have better performance for a particular workload or on
> + the hardware which wasn't available when the selection was made.
> + The environment variable, GLIBC_IFUNC=-xxx,yyy,-zzz...., can be
> + used to enable CPU/ARCH feature yyy, disable CPU/ARCH feature yyy
> + and zzz, where the feature name is case-sensitive and has to match
> + the ones in cpu-features.h. It can be used by glibc developers to
> + tune for a new processor or override the IFUNC selection to improve
> + performance for a particular workload.
> +
> + Since all CPU/ARCH features are hardware optimizations without
> + security implication, except for Prefer_MAP_32BIT_EXEC, which can
> + only be disabled, we check GLIBC_IFUNC for programs, including
> + set*id ones.
> +
> + NOTE: the IFUNC selection may change over time. Please check all
> + multiarch implementations when experimenting. */
> +
> + const char *p = valp->strval;
> + struct cpu_features *cpu_features = &GLRO(dl_x86_cpu_features);
> + const char *end = p + DEFAULT_STRLEN (p);
> + size_t len;
> +
> + do
> + {
> + const char *c, *n;
> + bool disable;
> + size_t nl;
> +
> + for (c = p; *c != ','; c++)
> + if (c >= end)
> + break;
> +
> + len = c - p;
> + disable = *p == '-';
> + if (disable)
> + {
> + n = p + 1;
> + nl = len - 1;
> + }
> + else
> + {
> + n = p;
> + nl = len;
> + }
> + switch (nl)
> + {
> + default:
> + break;
> + case 3:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX, 3);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CX8, 3);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA, 3);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, HTT, 3);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, RTM, 3);
> + }
> + break;
> + case 4:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX2, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI1, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, BMI2, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, CMOV, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, ERMS, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, FMA4, 4);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE2, 4);
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I586, 4);
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, I686, 4);
> + }
> + break;
> + case 5:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, LZCNT, 5);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, MOVBE, 5);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSSE3, 5);
> + }
> + break;
> + case 6:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, POPCNT, 6);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_1, 6);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, SSE4_2, 6);
> + }
> + break;
> + case 7:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512F, 7);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, OSXSAVE, 7);
> + }
> + break;
> + case 8:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512CD, 8);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512BW, 8);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512DQ, 8);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512ER, 8);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512PF, 8);
> + CHECK_GLIBC_IFUNC_CPU_OFF (n, cpu_features, AVX512VL, 8);
> + }
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Slow_BSF,
> + disable, 8);
> + break;
> + case 10:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX_Usable,
> + 10);
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA_Usable,
> + 10);
> + }
> + break;
> + case 11:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, AVX2_Usable,
> + 11);
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features, FMA4_Usable,
> + 11);
> + }
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Prefer_ERMS,
> + disable, 11);
> + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH (n, cpu_features,
> + Slow_SSE4_2, SSE4_2,
> + disable, 11);
> + break;
> + case 14:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features,
> + AVX512F_Usable, 14);
> + }
> + break;
> + case 15:
> + if (disable)
> + {
> + CHECK_GLIBC_IFUNC_ARCH_OFF (n, cpu_features,
> + AVX512DQ_Usable, 15);
> + }
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features, Fast_Rep_String,
> + disable, 15);
> + break;
> + case 16:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH
> + (n, cpu_features, Prefer_No_AVX512, AVX512F_Usable,
> + disable, 16);
> + }
> + break;
> + case 18:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> + Fast_Copy_Backward, disable,
> + 18);
> + }
> + break;
> + case 19:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> + Fast_Unaligned_Load, disable,
> + 19);
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> + Fast_Unaligned_Copy, disable,
> + 19);
> + }
> + break;
> + case 20:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH
> + (n, cpu_features, Prefer_No_VZEROUPPER, AVX_Usable,
> + disable, 20);
> + }
> + break;
> + case 21:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> + Prefer_MAP_32BIT_EXEC, disable,
> + 21);
> + }
> + break;
> + case 23:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_NEED_ARCH_BOTH
> + (n, cpu_features, AVX_Fast_Unaligned_Load, AVX_Usable,
> + disable, 23);
> + }
> + break;
> + case 26:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_NEED_CPU_BOTH
> + (n, cpu_features, Prefer_PMINUB_for_stringop, SSE2,
> + disable, 26);
> + }
> + break;
> + case 27:
> + {
> + CHECK_GLIBC_IFUNC_ARCH_BOTH (n, cpu_features,
> + Use_dl_runtime_resolve_slow,
> + disable, 27);
> + }
> + break;
> + }
> + p += len + 1;
> + }
> + while (p < end);
> +}
> +#endif
>