This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH 06/11] Add lock elision to rwlocks
- From: Torvald Riegel <triegel at redhat dot com>
- To: Andi Kleen <andi at firstfloor dot org>
- Cc: libc-alpha at sourceware dot org, Andi Kleen <ak at linux dot jf dot intel dot com>
- Date: Thu, 13 Jun 2013 18:28:22 +0200
- Subject: Re: [PATCH 06/11] Add lock elision to rwlocks
- References: <1370969416-8337-1-git-send-email-andi at firstfloor dot org> <1370969416-8337-7-git-send-email-andi at firstfloor dot org>
On Tue, 2013-06-11 at 09:50 -0700, Andi Kleen wrote:
> From: Andi Kleen <ak@linux.intel.com>
>
> This code is very similar to the main pthread mutex code, but we have
> to implement part of it in assembler.
>
> The patch kit is large because there are so many combinations
> of reader and writer locks, but it is fairly repetive.
>
> The basic elision algorithm is very similar to normal mutexes:
> a wrapper checks if the lock is free (both reader and writer)
> We have to check both reader and writer because separating them
> would require putting state on different cache lines.
We cannot use elision for trywrlock() or wrlock() because this would
change the semantics and violate POSIX semantics. See the guidelines.
Just using elision in rdlock and tryrdlock is okay.
>
> The algorithm is very basic currently and may be tuned
> more in the future.
>
> No adaptation at this point, the elision will not automatically
> disable itself. The programmer or user can do it explicitely though.
>
> New lock type flags are exposed to force enable/disable elision
> for a specific rwlocks.
>
> We use some padding in the internal rwlock as a elision flag.
> This is 8 bytes on 64bits currently, but could be made smaller later
> without compatibility problems (or to share some bytes with a
> adaptation count). 32bit uses one byte.
>
> 2013-05-16 Andi Kleen <ak@linux.intel.com>
> Hongjiu Lu <hongjiu.lu@intel.com>
>
> * pthread_rwlock_init.c (__pthread_rwlock_init): Support elision.
> * pthread_rwlockattr_setkind_np.c (pthread_rwlockattr_setkind_np): Support elision.
> * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S: dito.
> * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S: dito
> * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S: dito.
> * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S: dito.
> * sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S: dito.
> * sysdeps/unix/sysv/linux/internaltypes.h (struct pthread_condattr):
> (struct pthread_rwlockattr): Support elision flags in internal rwlock.
> * sysdeps/unix/sysv/linux/lowlevelrwlock.sym: Add ELIDED.
> * sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h: Add elision flags
> * sysdeps/unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c: New file
> * sysdeps/unix/sysv/linux/x86/pthread_rwlock_trywrlock.c: dito.
> * sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S: Handle elision.
> * sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedrdlock.S: dito.
> * sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_timedwrlock.S: dito.
> * sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_unlock.S: dito.
> * sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S: dito.
> ---
> nptl/pthread_rwlock_init.c | 4 ++
> nptl/pthread_rwlock_tryrdlock.c | 2 +
> nptl/pthread_rwlock_trywrlock.c | 2 +
> nptl/pthread_rwlockattr_setkind_np.c | 12 +++-
> .../sysv/linux/i386/i486/pthread_rwlock_rdlock.S | 68 ++++++++++++++++++++++
> .../linux/i386/i486/pthread_rwlock_timedrdlock.S | 63 ++++++++++++++++++++
> .../linux/i386/i486/pthread_rwlock_timedwrlock.S | 50 +++++++++++++++-
> .../sysv/linux/i386/i486/pthread_rwlock_unlock.S | 19 ++++++
> .../sysv/linux/i386/i486/pthread_rwlock_wrlock.S | 56 ++++++++++++++++++
> nptl/sysdeps/unix/sysv/linux/internaltypes.h | 14 ++++-
> nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym | 5 ++
> .../unix/sysv/linux/x86/bits/pthreadtypes.h | 6 +-
> .../unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c | 52 +++++++++++++++++
> .../unix/sysv/linux/x86/pthread_rwlock_trywrlock.c | 55 +++++++++++++++++
> .../unix/sysv/linux/x86_64/pthread_rwlock_rdlock.S | 51 ++++++++++++++++
> .../sysv/linux/x86_64/pthread_rwlock_timedrdlock.S | 51 ++++++++++++++++
> .../sysv/linux/x86_64/pthread_rwlock_timedwrlock.S | 35 +++++++++++
> .../unix/sysv/linux/x86_64/pthread_rwlock_unlock.S | 17 +++++-
> .../unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S | 34 +++++++++++
> 19 files changed, 589 insertions(+), 7 deletions(-)
> create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c
> create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_trywrlock.c
>
> diff --git a/nptl/pthread_rwlock_init.c b/nptl/pthread_rwlock_init.c
> index 29bef71..32d5a54 100644
> --- a/nptl/pthread_rwlock_init.c
> +++ b/nptl/pthread_rwlock_init.c
> @@ -68,6 +68,10 @@ __pthread_rwlock_init (rwlock, attr)
> header.private_futex));
> #endif
>
> +#ifdef __PTHREAD_RWLOCK_ELIDING
> + rwlock->__data.__eliding = iattr->eliding;
> +#endif
> +
> return 0;
> }
> strong_alias (__pthread_rwlock_init, pthread_rwlock_init)
> diff --git a/nptl/pthread_rwlock_tryrdlock.c b/nptl/pthread_rwlock_tryrdlock.c
> index 935ac87..896809b 100644
> --- a/nptl/pthread_rwlock_tryrdlock.c
> +++ b/nptl/pthread_rwlock_tryrdlock.c
> @@ -45,4 +45,6 @@ __pthread_rwlock_tryrdlock (pthread_rwlock_t *rwlock)
>
> return result;
> }
> +#ifndef __pthread_rwlock_tryrdlock
> strong_alias (__pthread_rwlock_tryrdlock, pthread_rwlock_tryrdlock)
> +#endif
> diff --git a/nptl/pthread_rwlock_trywrlock.c b/nptl/pthread_rwlock_trywrlock.c
> index 01754ae..6d8a8c4 100644
> --- a/nptl/pthread_rwlock_trywrlock.c
> +++ b/nptl/pthread_rwlock_trywrlock.c
> @@ -38,4 +38,6 @@ __pthread_rwlock_trywrlock (pthread_rwlock_t *rwlock)
>
> return result;
> }
> +#ifndef __pthread_rwlock_trywrlock
> strong_alias (__pthread_rwlock_trywrlock, pthread_rwlock_trywrlock)
> +#endif
> diff --git a/nptl/pthread_rwlockattr_setkind_np.c b/nptl/pthread_rwlockattr_setkind_np.c
> index 64bd341..833e1f0 100644
> --- a/nptl/pthread_rwlockattr_setkind_np.c
> +++ b/nptl/pthread_rwlockattr_setkind_np.c
> @@ -27,12 +27,22 @@ pthread_rwlockattr_setkind_np (attr, pref)
> {
> struct pthread_rwlockattr *iattr;
>
> + iattr = (struct pthread_rwlockattr *) attr;
> +
> + iattr->eliding = 0;
> + if (pref & (PTHREAD_RWLOCK_ELISION_NP|PTHREAD_RWLOCK_NO_ELISION_NP))
> + {
> + if (pref & PTHREAD_RWLOCK_ELISION_NP)
> + iattr->eliding = 1;
> + if (pref & PTHREAD_RWLOCK_NO_ELISION_NP)
> + iattr->eliding = -1;
> + pref &= ~(PTHREAD_RWLOCK_ELISION_NP|PTHREAD_RWLOCK_NO_ELISION_NP);
> + }
> if (pref != PTHREAD_RWLOCK_PREFER_READER_NP
> && pref != PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP
> && __builtin_expect (pref != PTHREAD_RWLOCK_PREFER_WRITER_NP, 0))
> return EINVAL;
>
> - iattr = (struct pthread_rwlockattr *) attr;
>
> iattr->lockkind = pref;
>
> diff --git a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S
> index 6c46ba6..62c95cb 100644
> --- a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S
> +++ b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_rdlock.S
> @@ -21,9 +21,16 @@
> #include <lowlevelrwlock.h>
> #include <pthread-errnos.h>
> #include <kernel-features.h>
> +#include <hle.h>
>
> #include <stap-probe.h>
>
> +#ifdef PIC
> +#define MO(x) x##@GOTOFF(%edx)
> +#else
> +#define MO(x) x
> +#endif
> +
> .text
>
> .globl __pthread_rwlock_rdlock
> @@ -43,6 +50,65 @@ __pthread_rwlock_rdlock:
>
> LIBC_PROBE (rdlock_entry, 1, %ebx)
>
> +#ifdef PIC
> + SETUP_PIC_REG(dx)
> + addl $_GLOBAL_OFFSET_TABLE_,%edx
> +#endif
> +
> + cmpl $0,MO(__elision_available)
> + jz not_elided_rdlock
> +
> + cmpb $0,ELIDING(%ebx)
> + js not_elided_rdlock
> + jnz 2f
> + /* zero: use default */
> +
> + cmpl $0,MO(__rwlock_rtm_enabled)
> + jz not_elided_rdlock
> +
> +2:
> + mov MO(__rwlock_rtm_read_retries),%ecx
> +
> +try_trans_rdlock:
> + XBEGIN abort_rdlock
> +
> + /* Lock writer/reader free? */
> + cmpl $0,WRITER(%ebx)
> + jnz 1f
> + cmpl $0,NR_READERS(%ebx)
> + jnz 1f
> +
> + /* Lock is free. Run with transaction */
> + xor %eax,%eax
> +
> + pop %ebx
> + cfi_adjust_cfa_offset(-4)
> + pop %esi
> + cfi_adjust_cfa_offset(-4)
> + ret
> +
> + /* Lock is not free. Run */
> +1: XABORT 0xff
> + jmp not_elided_rdlock
> +
> + /* Abort happened. */
> +abort_rdlock:
> + testl $_XABORT_CONFLICT,%eax
> + jz not_elided_rdlock
> +
> + /* For a reader that aborts due a conflict retry speculation
> + a limited number of times. This way when some reader aborts
> + because the reader count is written the other readers will
> + still elide, at the cost of retrying the speculation. */
> +
> + dec %ecx
> + jnz try_trans_rdlock
> +
> + /* Otherwise we just fall back directly to the lock.
> + Here's the place to add more adaptation. */
> +
> +not_elided_rdlock:
> +
> /* Get the lock. */
> movl $1, %edx
> xorl %eax, %eax
> @@ -188,5 +254,7 @@ __pthread_rwlock_rdlock:
> cfi_endproc
> .size __pthread_rwlock_rdlock,.-__pthread_rwlock_rdlock
>
> +#ifndef __pthread_rwlock_rdlock
> strong_alias (__pthread_rwlock_rdlock, pthread_rwlock_rdlock)
> hidden_def (__pthread_rwlock_rdlock)
> +#endif
> diff --git a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S
> index 1908f6f..053624d 100644
> --- a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S
> +++ b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedrdlock.S
> @@ -21,7 +21,13 @@
> #include <lowlevelrwlock.h>
> #include <pthread-errnos.h>
> #include <kernel-features.h>
> +#include <hle.h>
>
> +#ifdef PIC
> +#define MO(x) x##@GOTOFF(%edx)
> +#else
> +#define MO(x) x
> +#endif
>
> .text
>
> @@ -48,6 +54,62 @@ pthread_rwlock_timedrdlock:
> movl 28(%esp), %ebp
> movl 32(%esp), %edi
>
> +#ifdef PIC
> + SETUP_PIC_REG(dx)
> + addl $_GLOBAL_OFFSET_TABLE_,%edx
> +#endif
> +
> + cmpl $0,MO(__elision_available)
> + jz not_elided_trdlock
> +
> + cmpb $0,ELIDING(%ebp)
> + js not_elided_trdlock
> + jnz 2f
> + /* zero: use default */
> +
> + cmpl $0,MO(__rwlock_rtm_enabled)
> + jz not_elided_trdlock
> +
> +2:
> + mov MO(__rwlock_rtm_read_retries),%ecx
> +
> +try_trans_trdlock:
> + XBEGIN abort_trdlock
> +
> + /* Lock writer/reader free? */
> + cmpl $0,WRITER(%ebp)
> + jnz 1f
> + cmpl $0,NR_READERS(%ebp)
> + jnz 1f
> +
> + /* Lock is free. Run with transaction */
> + xor %eax,%eax
> + jmp 77f
> +
> +
> + /* Lock is not free. Run */
> +1: XABORT 0xff
> + jmp not_elided_trdlock
> +
> + /* Abort happened. */
> +abort_trdlock:
> + testl $_XABORT_CONFLICT,%eax
> + jz not_elided_trdlock
> +
> + /* For a reader that aborts due a conflict retry speculation
> + a limited number of times. This way when some reader aborts
> + because the reader count is written the other readers will
> + still elide, at the cost of retrying the speculation. */
> +
> + dec %ecx
> + jnz try_trans_trdlock
> +
> + /* Otherwise we just fall back directly to the lock.
> + Here's the place to add more adaptation. */
> +
> +not_elided_trdlock:
> +
> +
> /* Get the lock. */
> movl $1, %edx
> xorl %eax, %eax
> @@ -158,6 +220,7 @@ pthread_rwlock_timedrdlock:
>
> 7: movl %edx, %eax
>
> +77:
> addl $8, %esp
> cfi_adjust_cfa_offset(-8)
> popl %ebp
> diff --git a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S
> index e0fc809..6820395 100644
> --- a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S
> +++ b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_timedwrlock.S
> @@ -21,7 +21,13 @@
> #include <lowlevelrwlock.h>
> #include <pthread-errnos.h>
> #include <kernel-features.h>
> +#include <hle.h>
>
> +#ifdef PIC
> +#define MO(x) x##@GOTOFF(%edx)
> +#else
> +#define MO(x) x
> +#endif
>
> .text
>
> @@ -48,6 +54,48 @@ pthread_rwlock_timedwrlock:
> movl 28(%esp), %ebp
> movl 32(%esp), %edi
>
> +#ifdef PIC
> + SETUP_PIC_REG(dx)
> + addl $_GLOBAL_OFFSET_TABLE_,%edx
> +#endif
> +
> + cmpl $0,MO(__elision_available)
> + jz not_elided_twrlock
> +
> + cmpb $0,ELIDING(%ebp)
> + js not_elided_twrlock
> + jnz try_trans_wrlock
> + /* zero: use default */
> +
> + cmpl $0,MO(__rwlock_rtm_enabled)
> + jz not_elided_twrlock
> +
> +try_trans_wrlock:
> + XBEGIN abort_twrlock
> +
> + /* Lock writer free? */
> + cmpl $0,WRITER(%ebp)
> + jnz 1f
> + cmpl $0,NR_READERS(%ebp)
> + jnz 1f
> +
> + /* Lock is free. Run with transaction */
> + xor %eax,%eax
> +
> + jmp 77f
> +
> + /* Lock is not free. Run */
> +1: XABORT 0xff
> + jmp not_elided_twrlock
> +
> + /* Abort happened. */
> +abort_twrlock:
> +
> + /* Otherwise we just fall back directly to the lock.
> + Here's the place to add more adaptation. */
> +
> +not_elided_twrlock:
> +
> /* Get the lock. */
> movl $1, %edx
> xorl %eax, %eax
> @@ -156,7 +204,7 @@ pthread_rwlock_timedwrlock:
>
> 7: movl %edx, %eax
>
> - addl $8, %esp
> +77: addl $8, %esp
> cfi_adjust_cfa_offset(-8)
> popl %ebp
> cfi_adjust_cfa_offset(-4)
> diff --git a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S
> index 708e31c..3169743 100644
> --- a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S
> +++ b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_unlock.S
> @@ -20,6 +20,7 @@
> #include <lowlevellock.h>
> #include <lowlevelrwlock.h>
> #include <kernel-features.h>
> +#include <hle.h>
>
>
> .text
> @@ -38,6 +39,24 @@ __pthread_rwlock_unlock:
>
> movl 12(%esp), %edi
>
> + /* Is lock free? */
> + cmpl $0,WRITER(%edi)
> + jnz 1f
> + cmpl $0,NR_READERS(%edi)
> + jnz 1f
> +
> + /* Looks free. Assume transaction.
> + If you crash here you unlocked a free lock. */
> + XEND
> + xor %eax,%eax
> +
> + pop %edi
> + cfi_adjust_cfa_offset(-4)
> + pop %ebx
> + cfi_adjust_cfa_offset(-4)
> + ret
> +
> +1:
> /* Get the lock. */
> movl $1, %edx
> xorl %eax, %eax
> diff --git a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S
> index 6ea17f7..427ba34 100644
> --- a/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S
> +++ b/nptl/sysdeps/unix/sysv/linux/i386/i486/pthread_rwlock_wrlock.S
> @@ -21,9 +21,16 @@
> #include <lowlevelrwlock.h>
> #include <pthread-errnos.h>
> #include <kernel-features.h>
> +#include <hle.h>
>
> #include <stap-probe.h>
>
> +#ifdef PIC
> +#define MO(x) x@GOTOFF(%edx)
> +#else
> +#define MO(x) x
> +#endif
> +
> .text
>
> .globl __pthread_rwlock_wrlock
> @@ -43,6 +50,53 @@ __pthread_rwlock_wrlock:
>
> LIBC_PROBE (wrlock_entry, 1, %ebx)
>
> +#ifdef PIC
> + SETUP_PIC_REG(dx)
> + addl $_GLOBAL_OFFSET_TABLE_,%edx
> +#endif
> +
> + cmpl $0,MO(__elision_available)
> + jz not_elided_wrlock
> +
> + cmpb $0,ELIDING(%ebx)
> + js not_elided_wrlock
> + jnz try_trans_wrlock
> + /* zero: use default */
> +
> + cmpl $0,MO(__rwlock_rtm_enabled)
> + jz not_elided_wrlock
> +
> +try_trans_wrlock:
> + XBEGIN abort_wrlock
> +
> + /* Lock writer free? */
> + /* Ignore readers because we don't need them */
> + cmpl $0,WRITER(%ebx)
> + jnz 1f
> + cmpl $0,NR_READERS(%ebx)
> + jnz 1f
> +
> + /* Lock is free. Run with transaction */
> + xor %eax,%eax
> +
> + pop %ebx
> + cfi_adjust_cfa_offset(-4)
> + pop %esi
> + cfi_adjust_cfa_offset(-4)
> + ret
> +
> + /* Lock is not free. Run */
> +1: XABORT 0xff
> + jmp not_elided_wrlock
> +
> + /* Abort happened. */
> +abort_wrlock:
> +
> + /* Otherwise we just fall back directly to the lock.
> + Here's the place to add more adaptation. */
> +
> +not_elided_wrlock:
> +
> /* Get the lock. */
> movl $1, %edx
> xorl %eax, %eax
> @@ -179,5 +233,7 @@ __pthread_rwlock_wrlock:
> cfi_endproc
> .size __pthread_rwlock_wrlock,.-__pthread_rwlock_wrlock
>
> +#ifndef __pthread_rwlock_wrlock
> strong_alias (__pthread_rwlock_wrlock, pthread_rwlock_wrlock)
> hidden_def (__pthread_rwlock_wrlock)
> +#endif
> diff --git a/nptl/sysdeps/unix/sysv/linux/internaltypes.h b/nptl/sysdeps/unix/sysv/linux/internaltypes.h
> index 699a618..07ebb09 100644
> --- a/nptl/sysdeps/unix/sysv/linux/internaltypes.h
> +++ b/nptl/sysdeps/unix/sysv/linux/internaltypes.h
> @@ -20,7 +20,7 @@
> #define _INTERNALTYPES_H 1
>
> #include <stdint.h>
> -
> +#include <endian.h>
>
> struct pthread_attr
> {
> @@ -86,7 +86,17 @@ struct pthread_condattr
> struct pthread_rwlockattr
> {
> int lockkind;
> - int pshared;
> + /* This used to be an int pshared. We keep the pshared at the
> + same location, so that a one it in it earlier stays. */
Rephrase/formatting:
same location and the allowed values for this field,
PTHREAD_PROCESS_SHARED and PTHREAD_PROCESS_PRIVATE, both fit into one
byte. */
> +#if __BYTE_ORDER == __LITTLE_ENDIAN
> + short pshared;
> + char eliding;
> + char pad;
> +#else
> + char eliding;
> + char pad;
> + short pshared;
> +#endif
> };
>
>
> diff --git a/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym b/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym
> index f50b25b..fcbcd5a 100644
> --- a/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym
> +++ b/nptl/sysdeps/unix/sysv/linux/lowlevelrwlock.sym
> @@ -3,6 +3,10 @@
> #include <bits/pthreadtypes.h>
> #include <bits/wordsize.h>
>
> +#ifndef __PTHREAD_RWLOCK_ELIDING
> +#define __eliding __shared
> +#endif
> +
> --
>
> MUTEX offsetof (pthread_rwlock_t, __data.__lock)
> @@ -14,3 +18,4 @@ WRITERS_QUEUED offsetof (pthread_rwlock_t, __data.__nr_writers_queued)
> FLAGS offsetof (pthread_rwlock_t, __data.__flags)
> WRITER offsetof (pthread_rwlock_t, __data.__writer)
> PSHARED offsetof (pthread_rwlock_t, __data.__shared)
> +ELIDING offsetof (pthread_rwlock_t, __data.__eliding)
> diff --git a/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h b/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h
> index 1852e07..21b127d 100644
> --- a/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h
> +++ b/nptl/sysdeps/unix/sysv/linux/x86/bits/pthreadtypes.h
> @@ -183,12 +183,13 @@ typedef union
> unsigned int __nr_writers_queued;
> int __writer;
> int __shared;
> - unsigned long int __pad1;
> + long __eliding;
Call this __elision like in case of the mutexes.
If you choose a different name because of collision with the awful
__elision macro in 4/11, then that's another reason to change the macro,
not to pick another name here.
> unsigned long int __pad2;
> /* FLAGS must stay at this position in the structure to maintain
> binary compatibility. */
> unsigned int __flags;
> # define __PTHREAD_RWLOCK_INT_FLAGS_SHARED 1
> +# define __PTHREAD_RWLOCK_ELIDING 1
Can we call this __PTHREAD_RWLOCK_ELISION_ENABLED or something like
that?
> } __data;
> # else
> struct
> @@ -203,8 +204,9 @@ typedef union
> binary compatibility. */
> unsigned char __flags;
> unsigned char __shared;
> - unsigned char __pad1;
> + char __eliding;
See above.
> unsigned char __pad2;
> +# define __PTHREAD_RWLOCK_ELIDING 2
> int __writer;
> } __data;
> # endif
> diff --git a/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c b/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c
> new file mode 100644
> index 0000000..0bee834
> --- /dev/null
> +++ b/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_tryrdlock.c
> @@ -0,0 +1,52 @@
> +/* pthread_rwlock_tryrdlock: Lock eliding version of pthreads rwlock_tryrdlock.
> + Copyright (C) 2013 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +#include <pthread.h>
> +#include <pthreadP.h>
> +#include <hle.h>
> +#include <elision-conf.h>
> +#include "init-arch.h"
> +
> +#define __pthread_rwlock_tryrdlock __full_pthread_rwlock_tryrdlock
> +#include <nptl/pthread_rwlock_tryrdlock.c>
> +#undef __pthread_rwlock_tryrdlock
> +
> +int
> +__pthread_rwlock_tryrdlock(pthread_rwlock_t *rwlock)
> +{
> + unsigned status;
> +
> + if ((rwlock->__data.__eliding > 0 && __elision_available)
> + || (rwlock->__data.__eliding == 0 && __rwlock_rtm_enabled))
> + {
> + if ((status = _xbegin()) == _XBEGIN_STARTED)
> + {
> + if (rwlock->__data.__writer == 0
> + && rwlock->__data.__nr_readers == 0)
> + return 0;
> + /* Lock was busy. Fall back to normal locking.
> + Could also _xend here but xabort with 0xff code
> + is more visible in the profiler. */
whitespace
> + _xabort (0xff);
> + }
> + /* Aborts come here */
> + }
> +
> + return __full_pthread_rwlock_tryrdlock (rwlock);
> +}
> +
> +strong_alias(__pthread_rwlock_tryrdlock, pthread_rwlock_tryrdlock);
> diff --git a/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_trywrlock.c b/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_trywrlock.c
> new file mode 100644
> index 0000000..10411b8
> --- /dev/null
> +++ b/nptl/sysdeps/unix/sysv/linux/x86/pthread_rwlock_trywrlock.c
> @@ -0,0 +1,55 @@
> +/* pthread_rwlock_trywrlock: Lock eliding version of pthreads rwlock trywrlock.
> + Copyright (C) 2013 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +#include <pthread.h>
> +#include <pthreadP.h>
> +#include <hle.h>
> +#include <elision-conf.h>
> +#include "init-arch.h"
> +
> +#define __pthread_rwlock_trywrlock __full_pthread_rwlock_trywrlock
> +#include <nptl/pthread_rwlock_trywrlock.c>
> +#undef __pthread_rwlock_trywrlock
> +
> +int
> +__pthread_rwlock_trywrlock(pthread_rwlock_t *rwlock)
> +{
> + unsigned status;
> + int elision = 0;
> +
> + if (rwlock->__data.__eliding == 0 && __rwlock_rtm_enabled)
> + {
> + _xabort (0xfd);
> + elision = 1;
Is _xabort a NOP if we're not executing speculatively? If so, this
needs an explanation; it's far from obvious how this works.
> + }
> + if (elision || (rwlock->__data.__eliding > 0 && __elision_available))
> + {
> + if ((status = _xbegin()) == _XBEGIN_STARTED)
> + {
> + if (rwlock->__data.__writer == 0
> + && rwlock->__data.__nr_readers == 0)
> + return 0;
> + /* Lock was busy. Fall back to normal locking.
> + Could also _xend here but xabort with 0xff code
> + is more visible in the profiler. */
> + _xabort (0xff);
> + }
> + }
We cannot use elision for write locks without changing the semantics.
See the guidelines.
Same applies to the other write lock routines.