This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[PATCH 07/15] Add the low level infrastructure for pthreads lock elision with TSX


From: Andi Kleen <ak@linux.intel.com>

Lock elision using TSX is a technique to optimize lock scaling
It allows to run locks in parallel using hardware support for
a transactional execution mode in upcoming Intel CPUs.
See http://software.intel.com/file/41604 for the full
specification.

This patch implements a simple adaptive lock elision algorithm based
on RTM. It enables elision for the pthread mutexes and rwlocks.
The algorithm keeps track whether a mutex successfully elides or not,
and stops eliding for some time when it is not.

When the CPU supports RTM the elision path is automatically tried,
otherwise any elision is disabled.

The adaptation algorithm and its tuning is currently preliminary.

The code adds some checks to the lock fast paths.  IFUNC is used
to select functions with these checks based on the RTM availability.

All the parameters of the algorithms are tunable using environment
variables.

Lock elision can be enabled/disabled using environment variables.
It can be also enabled or disabled using new lock types for
mutex and rwlocks.

This patch implements the low level "lll_" code for lock elision.
Followon patches hook this into the pthread implement.

The IFUNC implementation currently triggers some linker bugs,
that show up in the test suite when libc calls locks internally.
It works with HJ Lu's latest linker, but other linkers may
show additional failures in the test suite.
Linker known to work:
http://www.kernel.org/pub/linux/devel/binutils/binutils-2.23.51.0.8.tar.xz

Changes with the RTM mutexes:
-----------------------------
Lock elision in pthreads is generally compatible with existing programs.
There are some obscure exceptions, which are expected to be uncommon.
See the manual for more details.

- A broken program that unlocks a free lock will crash.
  There are ways around this with some tradeoffs (more code in hot paths)
  This will also happen on systems without RTM with the patchkit.
  I'm still undecided on what approach to take here; have to wait for testing reports.
- pthread_mutex_destroy of a lock mutex will not return EBUSY but 0.
- mutex appears free when elided.
  pthread_mutex_lock(mutex);
  if (pthread_mutex_trylock(mutex) != 0) do_something
  will not do something when the lock elided.
  However note that if the check is an assert it works as expected because the
  assert failure aborts and the region is re-executed non transactionally,
  with the old behaviour.
- There's also a similar situation with trylock outside the mutex,
  "knowing" that the mutex must be held due to some other condition.
  In this case an assert failure cannot be recovered. This situation is
  usually an existing bug in the program.
- Same applies to the rwlocks. Some of the return values changes
  (for example there is no EDEADLK for an elided lock, unless it aborts.
   However when elided it will also never deadlock of course)
- Timing changes, so broken programs that make assumptions about specific timing
  may expose already existing latent problems.  Note that these broken programs will
  break in other situations too (loaded system, new faster hardware, compiler
  optimizations etc.)

Currently elision is enabled by default on systems that support RTM,
unless explicitely disabled either in the program or by the user.
Given more experience we can decide if that is a good idea, or if it
should be opt-in.

This patch implements the basic infrastructure for elision.

Open issues:
- XTEST or not XTEST in unlock, see above.
- Find workarounds for the linker bugs
- There's a obscure race when first setting the elision type
the first time.
- Adaptation for rwlocks
- Condition variables don't use elision so far
- Better adaptation algorithms
- Elide spinlocks too?

2012-12-22  Andi Kleen  <ak@linux.intel.com>
            Hongjiu Lu <hongjiu.lu@intel.com>

	* nptl/nptl-init.c (__pthread_force_elision): Add.
	* nptl/pthreadP.h (__pthread_force_elision): Add.
	* nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h (__lll_timedwait_tid,
          lll_timedlock_elision, __lll_lock_elision, __lll_unlock_elision,
          __lll_trylock_elision, lll_lock_elision, lll_unlock_elision,
	  lll_trylock_elision): Add.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/Makefile: Imply x86
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.c: New file.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.h: New file.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-lock.c: New file.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-timed.c: New file.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-trylock.c: New file.
	* nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-unlock.c: New file
	* nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h (__lll_timedwait_tid,
          lll_timedlock_elision, __lll_lock_elision, __lll_unlock_elision,
          __lll_trylock_elision, lll_lock_elision, lll_unlock_elision,
	  lll_trylock_elision): Add.
	* nptl/elision-conf.h: New file.
---
 nptl/elision-conf.h                                |    1 +
 nptl/nptl-init.c                                   |    1 +
 nptl/pthreadP.h                                    |    2 +
 nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h   |   22 ++
 .../sysdeps/unix/sysv/linux/x86/multiarch/Makefile |    3 +
 .../unix/sysv/linux/x86/multiarch/elision-conf.c   |  243 ++++++++++++++++++++
 .../unix/sysv/linux/x86/multiarch/elision-conf.h   |   54 +++++
 .../unix/sysv/linux/x86/multiarch/elision-lock.c   |   94 ++++++++
 .../unix/sysv/linux/x86/multiarch/elision-timed.c  |    8 +
 .../sysv/linux/x86/multiarch/elision-trylock.c     |   64 +++++
 .../unix/sysv/linux/x86/multiarch/elision-unlock.c |   32 +++
 nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h |   23 ++
 12 files changed, 547 insertions(+), 0 deletions(-)
 create mode 100644 nptl/elision-conf.h
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/Makefile
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.c
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.h
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-lock.c
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-timed.c
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-trylock.c
 create mode 100644 nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-unlock.c

diff --git a/nptl/elision-conf.h b/nptl/elision-conf.h
new file mode 100644
index 0000000..40a8c17
--- /dev/null
+++ b/nptl/elision-conf.h
@@ -0,0 +1 @@
+/* empty */
diff --git a/nptl/nptl-init.c b/nptl/nptl-init.c
index 6a18dbe..4e29f64 100644
--- a/nptl/nptl-init.c
+++ b/nptl/nptl-init.c
@@ -36,6 +36,7 @@
 #include <lowlevellock.h>
 #include <kernel-features.h>
 
+int __pthread_force_elision attribute_hidden;
 
 /* Size and alignment of static TLS block.  */
 size_t __static_tls_size;
diff --git a/nptl/pthreadP.h b/nptl/pthreadP.h
index 5cff300..6e1902e 100644
--- a/nptl/pthreadP.h
+++ b/nptl/pthreadP.h
@@ -573,6 +573,8 @@ extern void __free_stacks (size_t limit) attribute_hidden;
 
 extern void __wait_lookup_done (void) attribute_hidden;
 
+extern int __pthread_force_elision attribute_hidden;
+
 #ifdef SHARED
 # define PTHREAD_STATIC_FN_REQUIRE(name)
 #else
diff --git a/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h b/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h
index 7b576ca..b9bf2a6 100644
--- a/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h
+++ b/nptl/sysdeps/unix/sysv/linux/i386/lowlevellock.h
@@ -429,6 +429,12 @@ LLL_STUB_UNWIND_INFO_END
 		       : "memory");					      \
      result; })
 
+extern int __lll_timedlock_elision (int *futex, short *try_lock,
+					 const struct timespec *timeout,
+					 int private) attribute_hidden;
+
+#define lll_timedlock_elision(futex, try_lock, timeout, private)	\
+  __lll_timedlock_elision(&(futex), &(try_lock), timeout, private)
 
 #define lll_robust_timedlock(futex, timeout, id, private) \
   ({ int result, ignore1, ignore2, ignore3;				      \
@@ -582,6 +588,22 @@ extern int __lll_timedwait_tid (int *tid, const struct timespec *abstime)
       }									      \
     __result; })
 
+extern int __lll_lock_elision (int *futex, short *try_lock, int private)
+  attribute_hidden;
+
+extern int __lll_unlock_elision(int *lock, int private)
+  attribute_hidden;
+
+extern int __lll_trylock_elision(int *lock, short *try_lock)
+  attribute_hidden;
+
+#define lll_lock_elision(futex, try_lock, private) \
+  __lll_lock_elision (&(futex), &(try_lock), private)
+#define lll_unlock_elision(futex, private) \
+  __lll_unlock_elision (&(futex), private)
+#define lll_trylock_elision(futex, try_lock) \
+  __lll_trylock_elision(&(futex), &(try_lock))
+
 #endif  /* !__ASSEMBLER__ */
 
 #endif	/* lowlevellock.h */
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/Makefile b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/Makefile
new file mode 100644
index 0000000..61b7552
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/Makefile
@@ -0,0 +1,3 @@
+libpthread-sysdep_routines += init-arch
+libpthread-sysdep_routines += elision-lock elision-unlock elision-timed \
+			      elision-trylock
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.c b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.c
new file mode 100644
index 0000000..479ace4
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.c
@@ -0,0 +1,243 @@
+/* elision-conf.c: Lock elision tunable parameters.
+   Copyright (C) 2012 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>. */
+#include <pthreadP.h>
+#include <sys/fcntl.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <init-arch.h>
+#include "elision-conf.h"
+
+struct elision_config __elision_aconf =
+  {
+    .retry_lock_busy = 3,
+    .retry_lock_internal_abort = 3,
+    .retry_try_xbegin = 3,
+    .retry_trylock_internal_abort = 3,
+  };
+
+struct tune
+{
+  const char *name;
+  unsigned offset;
+  int len;
+};
+
+#define FIELD(x) { #x, offsetof(struct elision_config, x), sizeof(#x)-1 }
+
+static const struct tune tunings[] =
+  {
+    FIELD(retry_lock_busy),
+    FIELD(retry_lock_internal_abort),
+    FIELD(retry_try_xbegin),
+    FIELD(retry_trylock_internal_abort),
+    {}
+  };
+
+#define PAIR(x) x, sizeof (x)-1
+
+/* It's dangerous to reference anything else here due to IFUNC requirements,
+   so we implement all the string functions we need ourself. */
+
+static int
+simple_strncmp (const char *a, const char *b, int len)
+{
+  int i;
+  for (i = 0; i < len; i++)
+    {
+      if (*a != *b)
+        return *a - *b;
+      if (*a++ == 0 || *b++ == 0)
+        break;
+    }
+  return 0;
+}
+
+static int
+simple_strtou (const char *s, char **end)
+{
+  unsigned num = 0;
+
+  while (*s >= '0' && *s <= '9')
+    num = (num * 10) + *s++ - '0';
+  if (end)
+    *(const char **)end = s;
+  return num;
+}
+
+static void
+complain (const char *msg, int len)
+{
+  INTERNAL_SYSCALL_DECL (err);
+  INTERNAL_SYSCALL (write, err, 3, 2, (char *)msg, len);
+}
+
+static void
+elision_aconf_setup(const char *s)
+{
+  int i;
+
+  while (*s)
+    {
+      for (i = 0; tunings[i].name; i++)
+	{
+	  int nlen = tunings[i].len;
+
+	  if (!simple_strncmp (tunings[i].name, s, nlen) && s[nlen] == ':')
+	    {
+	      char *end;
+	      int val;
+
+	      s += nlen + 1;
+	      val = simple_strtou (s, &end);
+	      if (end == s)
+		goto error;
+	      *(int *)(((char *)&__elision_aconf) + tunings[i].offset) = val;
+	      s = end;
+	      if (*s == ',')
+		s++;
+	      else if (*s)
+		goto error;
+	    }
+	}
+    }
+  return;
+
+ error:
+  complain (PAIR("pthreads: invalid PTHREAD_MUTEX syntax\n"));
+}
+
+int __rwlock_rtm_enabled attribute_hidden;
+int __rwlock_rtm_read_retries attribute_hidden = 3;
+int __elision_available attribute_hidden;
+
+#define PAIR(x) x, sizeof (x)-1
+
+static char *
+next_env_entry (char first, char ***position)
+{
+  char **current = *position;
+  char *result = NULL;
+
+  while (*current != NULL)
+    {
+      if ((*current)[0] == first)
+	{
+	  result = *current;
+	  *position = ++current;
+	  break;
+	}
+
+      ++current;
+    }
+
+  return result;
+}
+
+static inline void
+match (const char *line, const char *var, int len, const char **res)
+{
+  if (!simple_strncmp (line, var, len))
+    *res = line + len;
+}
+
+static void
+elision_mutex_init (const char *s)
+{
+  if (!s)
+    return;
+  if (!simple_strncmp (s, "adaptive", 8) && (s[8] == 0 || s[8] == ':'))
+    {
+      __pthread_force_elision = __elision_available;
+      if (s[8] == ':')
+	elision_aconf_setup (s + 9);
+    }
+  else if (!simple_strncmp (s, "elision", 7) && (s[7] == 0 || s[7] == ':'))
+    {
+      __pthread_force_elision = __elision_available;
+      if (s[7] == ':')
+        elision_aconf_setup (s + 8);
+    }
+  else if (!simple_strncmp (s, "none", 4) && s[4] == 0)
+    __pthread_force_elision = 0;
+  else
+    complain (PAIR("pthreads: Unknown setting for PTHREAD_MUTEX\n"));
+}
+
+static void
+elision_rwlock_init (const char *s)
+{
+  if (!s)
+    {
+      __rwlock_rtm_enabled = __elision_available;
+      return;
+    }
+  if (!simple_strncmp (s, "elision", 7))
+    {
+      __rwlock_rtm_enabled = __elision_available;
+      if (s[7] == ':')
+        {
+          char *end;
+	  int n;
+
+          n = simple_strtou (s + 8, &end);
+	  if (end == s + 8)
+	    complain (PAIR("pthreads: Bad retry number for PTHREAD_RWLOCK\n"));
+          else
+	    __rwlock_rtm_read_retries = n;
+	}
+    }
+  else if (!simple_strncmp(s, "none", 4) && s[4] == 0)
+    __rwlock_rtm_enabled = 0;
+  else
+    complain (PAIR("pthreads: Unknown setting for PTHREAD_RWLOCK\n"));
+}
+
+static void
+elision_init (int argc __attribute__ ((unused)),
+	      char **argv  __attribute__ ((unused)),
+	      char **environ)
+{
+  char *envline;
+  const char *mutex = NULL, *rwlock = NULL;
+
+  __pthread_force_elision = 1;
+  __elision_available = 1;
+
+  while ((envline = next_env_entry ('P', &environ)) != NULL)
+    {
+      match (envline, PAIR("PTHREAD_MUTEX="), &mutex);
+      match (envline, PAIR("PTHREAD_RWLOCK="), &rwlock);
+    }
+
+  elision_mutex_init (mutex);
+  elision_rwlock_init (rwlock);
+}
+
+#ifdef SHARED
+# define INIT_SECTION ".init_array"
+#else
+# define INIT_SECTION ".preinit_array"
+#endif
+
+void (*const init_array []) (int, char **, char **)
+  __attribute__ ((section (INIT_SECTION), aligned (sizeof (void *)))) =
+{
+  &elision_init
+};
+
+__pthread_abort_hook_t __tsx_abort_hook attribute_hidden;
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.h b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.h
new file mode 100644
index 0000000..f6d07ff
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-conf.h
@@ -0,0 +1,54 @@
+/* elision-conf.h: Lock elision tunable parameters.
+   Copyright (C) 2012 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>. */
+#ifndef _ELISION_CONF_H
+#define _ELISION_CONF_H 1
+
+#include <pthread.h>
+#include <cpuid.h>
+#include <time.h>
+
+/* Should make sure there is no false sharing on this */
+
+struct elision_config
+{
+  int retry_lock_busy;
+  int retry_lock_internal_abort;
+  int retry_try_xbegin;
+  int retry_trylock_internal_abort;
+};
+
+extern struct elision_config __elision_aconf attribute_hidden;
+
+extern __pthread_abort_hook_t __tsx_abort_hook;
+
+extern int __rwlock_rtm_enabled;
+extern int __elision_available;
+
+extern int __pthread_mutex_timedlock_nortm (pthread_mutex_t *mutex, const struct timespec *);
+extern int __pthread_mutex_timedlock_rtm (pthread_mutex_t *mutex, const struct timespec *);
+extern int __pthread_mutex_timedlock (pthread_mutex_t *mutex, const struct timespec *);
+extern int __pthread_mutex_lock_nortm (pthread_mutex_t *mutex);
+extern int __pthread_mutex_lock_rtm (pthread_mutex_t *mutex);
+extern int __pthread_mutex_lock (pthread_mutex_t *mutex);
+extern int __pthread_mutex_trylock_nortm (pthread_mutex_t *);
+extern int __pthread_mutex_trylock_rtm (pthread_mutex_t *);
+extern int __pthread_mutex_trylock (pthread_mutex_t *);
+
+#define SUPPORTS_ELISION 1
+
+#endif
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-lock.c b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-lock.c
new file mode 100644
index 0000000..83b82b8
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-lock.c
@@ -0,0 +1,94 @@
+/* elision-lock.c: Elided pthread mutex lock.
+   Copyright (C) 2011, 2012 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>. */
+#include <pthread.h>
+#include "pthreadP.h"
+#include "lowlevellock.h"
+#include "hle.h"
+#include "elision-conf.h"
+
+#if !defined(LLL_LOCK) && !defined(EXTRAARG)
+/* Make sure the configuration code is always linked in for static
+   libraries. */
+#include "elision-conf.c"
+#endif
+
+#ifndef EXTRAARG
+#define EXTRAARG
+#endif
+#ifndef LLL_LOCK
+#define LLL_LOCK(a,b) lll_lock(a,b), 0
+#endif
+
+#define aconf __elision_aconf
+
+/* Adaptive lock using transactions.
+   By default the lock region is run as a transaction, and when it
+   aborts or the lock is busy the lock adapts itself. */
+
+int
+__lll_lock_elision (int *futex, short *try_lock, EXTRAARG int private)
+{
+  if (*try_lock <= 0)
+    {
+      unsigned status;
+      int try_xbegin;
+
+      for (try_xbegin = aconf.retry_try_xbegin;
+	   try_xbegin > 0;
+	   try_xbegin--)
+	{
+	  if ((status = _xbegin()) == _XBEGIN_STARTED)
+	    {
+	      if (*futex == 0)
+		return 0;
+
+	      /* Lock was busy. Fall back to normal locking.
+		 Could also _xend here but xabort with 0xff code
+		 is more visible in the profiler. */
+	      _xabort (0xff);
+
+	      if (*try_lock != aconf.retry_lock_busy)
+		*try_lock = aconf.retry_lock_busy;
+	      break;
+	    }
+
+
+	  if (__tsx_abort_hook)
+	    __tsx_abort_hook(status);
+
+	  if (!(status & _XABORT_RETRY))
+	    {
+	      /* Internal abort. There is no chance for retry.
+		 Use the normal locking and next time use lock.
+		 Be careful to avoid writing to the lock. */
+	      if (*try_lock != aconf.retry_lock_internal_abort)
+		*try_lock = aconf.retry_lock_internal_abort;
+	      break;
+	    }
+	}
+    }
+  else
+    {
+      /* Use a normal lock until the threshold counter runs out.
+	 Lost updates possible. */
+      (*try_lock)--;
+    }
+
+  /* Use a normal lock here */
+  return LLL_LOCK ((*futex), private);
+}
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-timed.c b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-timed.c
new file mode 100644
index 0000000..1cad4779
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-timed.c
@@ -0,0 +1,8 @@
+#include <time.h>
+#include "elision-conf.h"
+#include "lowlevellock.h"
+#define __lll_lock_elision __lll_timedlock_elision
+#define EXTRAARG const struct timespec *t,
+#undef LLL_LOCK
+#define LLL_LOCK(a, b) lll_timedlock(a, t, b)
+#include "elision-lock.c"
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-trylock.c b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-trylock.c
new file mode 100644
index 0000000..c2f8113
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-trylock.c
@@ -0,0 +1,64 @@
+/* elision-trylock.c: Lock eliding trylock for pthreads.
+   Copyright (C) 2011 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>. */
+#include <pthread.h>
+#include <pthreadP.h>
+#include <lowlevellock.h>
+#include "hle.h"
+#include "elision-conf.h"
+
+#define aconf __elision_aconf
+
+int
+__lll_trylock_elision (int *futex, short *try_lock)
+{
+  /* Only try a transaction if it's worth it */
+  if (*try_lock <= 0)
+    {
+      unsigned status;
+
+      if ((status = _xbegin()) == _XBEGIN_STARTED)
+	{
+	  if (*futex == 0)
+	    return 0;
+
+	  /* Lock was busy. Fall back to normal locking.
+	     Could also _xend here but xabort with 0xff code
+	     is more visible in the profiler. */
+	  _xabort (0xff);
+	}
+
+
+      if (__tsx_abort_hook)
+	__tsx_abort_hook (status);
+
+      if (!(status & _XABORT_RETRY))
+        {
+          /* Internal abort. No chance for retry. For future
+             locks don't try speculation for some time. */
+          if (*try_lock != aconf.retry_trylock_internal_abort)
+            *try_lock = aconf.retry_trylock_internal_abort;
+        }
+    }
+  else
+    {
+      /* Lost updates are possible, but harmless. */
+      (*try_lock)--;
+    }
+
+  return lll_trylock (*futex);
+}
diff --git a/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-unlock.c b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-unlock.c
new file mode 100644
index 0000000..49bdf5e
--- /dev/null
+++ b/nptl/sysdeps/unix/sysv/linux/x86/multiarch/elision-unlock.c
@@ -0,0 +1,32 @@
+/* elision-unlock.c: Commit an elided pthread lock.
+   Copyright (C) 2012 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+#include "pthreadP.h"
+#include "lowlevellock.h"
+#include "hle.h"
+
+int
+__lll_unlock_elision(int *lock, int private)
+{
+  /* When the lock was free we're in a transaction.
+     When you crash here you unlocked a free lock. */
+  if (*lock == 0)
+    _xend();
+  else
+    lll_unlock ((*lock), private);
+  return 0;
+}
diff --git a/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h b/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
index 5a80ddd..20198b4 100644
--- a/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
+++ b/nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.h
@@ -426,6 +426,13 @@ LLL_STUB_UNWIND_INFO_END
 		       : "memory", "cx", "cc", "r10", "r11");		      \
      result; })
 
+extern int __lll_timedlock_elision (int *futex, short *try_lock,
+					 const struct timespec *timeout,
+					 int private) attribute_hidden;
+
+#define lll_timedlock_elision(futex, try_lock, timeout, private)	\
+  __lll_timedlock_elision(&(futex), &(try_lock), timeout, private)
+
 #define lll_robust_timedlock(futex, timeout, id, private) \
   ({ int result, ignore1, ignore2, ignore3;				      \
      __asm __volatile (LOCK_INSTR "cmpxchgl %1, %4\n\t"			      \
@@ -596,6 +603,22 @@ extern int __lll_timedwait_tid (int *tid, const struct timespec *abstime)
       }									      \
     __result; })
 
+extern int __lll_lock_elision (int *futex, short *try_lock, int private)
+  attribute_hidden;
+
+extern int __lll_unlock_elision(int *lock, int private)
+  attribute_hidden;
+
+extern int __lll_trylock_elision(int *lock, short *try_lock)
+  attribute_hidden;
+
+#define lll_lock_elision(futex, try_lock, private) \
+  __lll_lock_elision (&(futex), &(try_lock), private)
+#define lll_unlock_elision(futex, private) \
+  __lll_unlock_elision (&(futex), private)
+#define lll_trylock_elision(futex, try_lock) \
+  __lll_trylock_elision(&(futex), &(try_lock))
+
 #endif  /* !__ASSEMBLER__ */
 
 #endif	/* lowlevellock.h */
-- 
1.7.7.6


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]