[RFC] [PATCH 3] uprobes: return probes

[Jim Keniston <jkenisto at us dot ibm dot com>]

This patch implements user-space return probes (uretprobes) atop
the two uprobes patches I posted on Friday -- e.g.,
http://sources.redhat.com/ml/systemtap/2007-q2/msg00108.html

The big difference from previous implementations of uretprobes is that
the API function init_uretprobes() is gone.  Uprobes uses slot 0 of
the SSOL area for the uretprobe trampoline, so you no longer have to
figure out where to put the trampoline.

Comments welcome.

Jim Keniston

This patch implements user-space return probes (uretprobes). Similar to
kretprobes, uretprobes works by bouncing a probed function's return off
a known trampoline, at which time the user-specified handler is run.

Uretprobes works by first inserting a uprobe at the entry to the
specified function.  When the function is called and the probe is hit,
uretprobes makes a copy of the return address, then replaces the
return address with the address of the trampoline.

When the function returns, control passes to the trampoline and
then to uprobes.  Uprobes runs the user-specified handler, then
allows the thread to continue at the real return address.

Uprobes uses one slot of the SSOL area for the trampoline.

---

 Documentation/uprobes.txt  |  247 +++++++++++++++++++++++++++++++++++++++-----
 arch/i386/kernel/uprobes.c |   40 +++++++
 include/asm-i386/uprobes.h |   12 +-
 include/linux/uprobes.h    |   43 +++++++
 kernel/uprobes.c           |  250 +++++++++++++++++++++++++++++++++++++++++++--
 5 files changed, 553 insertions(+), 39 deletions(-)

diff -puN Documentation/uprobes.txt~3-uretprobes Documentation/uprobes.txt
--- linux-2.6.21-rc6/Documentation/uprobes.txt~3-uretprobes	2007-04-23 15:30:29.000000000 -0700
+++ linux-2.6.21-rc6-jimk/Documentation/uprobes.txt	2007-04-23 16:26:43.000000000 -0700
@@ -3,7 +3,7 @@ Author	: Jim Keniston <jkenisto@us.ibm.c
 
 CONTENTS
 
-1. Concepts
+1. Concepts: Uprobes, Return Probes
 2. Architectures Supported
 3. Configuring Uprobes
 4. API Reference
@@ -14,15 +14,22 @@ CONTENTS
 9. TODO
 10. Uprobes Team
 11. Uprobes Example
+12. Uretprobes Example
 
-1. Concepts
+1. Concepts: Uprobes, Return Probes
 
 Uprobes enables you to dynamically break into any routine in a
 user application and collect debugging and performance information
 non-disruptively. You can trap at any code address, specifying a
 kernel handler routine to be invoked when the breakpoint is hit.
 
-The registration function, register_uprobe(), specifies which
+There are currently two types of user-space probes: uprobes and
+uretprobes (also called return probes).  A uprobe can be inserted on
+any instruction in the application's virtual address space.  A return
+probe fires when a specified user function returns.  These two probe
+types are discussed in more detail later.
+
+A registration function such as register_uprobe() specifies which
 process is to be probed, where the probe is to be inserted, and what
 handler is to be called when the probe is hit.
 
@@ -62,6 +69,10 @@ remove the breakpoint instruction.  This
 multithreaded application.  For example, it would open a time window
 when another thread could sail right past the probepoint.)
 
+Instruction copies to be single-stepped are stored in a per-process
+"single-step out of line (SSOL) area," which is a little VM area
+created by Uprobes in each probed process's address space.
+
 1.2 The Role of Utrace
 
 When a probe is registered on a previously unprobed process,
@@ -73,7 +84,23 @@ Uprobes of breakpoint and single-step tr
 events in the lifetime of the probed process, such as fork, clone,
 exec, and exit.
 
-1.3 Multithreaded Applications
+1.3 How Does a Return Probe Work?
+
+When you call register_uretprobe(), Uprobes establishes a uprobe
+at the entry to the function.  When the probed function is called
+and this probe is hit, Uprobes saves a copy of the return address,
+and replaces the return address with the address of a "trampoline"
+-- a piece of code that contains a breakpoint instruction.
+
+When the probed function executes its return instruction, control
+passes to the trampoline and that breakpoint is hit.  Uprobes's
+trampoline handler calls the user-specified handler associated with the
+uretprobe, then sets the saved instruction pointer to the saved return
+address, and that's where execution resumes upon return from the trap.
+
+The trampoline is stored in the SSOL area.
+
+1.4 Multithreaded Applications
 
 Uprobes supports the probing of multithreaded applications.  Uprobes
 imposes no limit on the number of threads in a probed application.
@@ -95,7 +122,8 @@ after the breakpoint has been inserted/r
 
 2. Architectures Supported
 
-Uprobes is implemented on the following architectures:
+Uprobes and uretprobes are implemented on the following
+architectures:
 
 - i386
 - x86_64 (AMD-64, EM64T)	// in progress
@@ -119,10 +147,10 @@ unloading" (CONFIG_MODULE_UNLOAD) are se
 
 4. API Reference
 
-The Uprobes API includes two functions, register_uprobe() and
-unregister_uprobe().  Here are terse, mini-man-page specifications for
-these functions and the associated probe handlers that you'll write.
-See the latter half of this document for an example.
+The Uprobes API includes a "register" function and an "unregister"
+function for each type of probe.  Here are terse, mini-man-page
+specifications for these functions and the associated probe handlers
+that you'll write.  See the latter half of this document for examples.
 
 4.1 register_uprobe
 
@@ -143,12 +171,41 @@ Called with u pointing to the uprobe ass
 and regs pointing to the struct containing the registers saved when
 the breakpoint was hit.
 
-4.2 unregister_uprobe
+4.2 register_uretprobe
+
+#include <linux/uprobes.h>
+int register_uretprobe(struct uretprobe *rp);
+
+Establishes a return probe in the process whose pid is rp->u.pid for
+the function whose address is rp->u.vaddr.  When that function returns,
+Uprobes calls rp->handler.
+
+register_uretprobe() returns 0 on success, or a negative errno
+otherwise.
+
+User's return-probe handler (rp->handler):
+#include <linux/uprobes.h>
+#include <linux/ptrace.h>
+void uretprobe_handler(struct uretprobe_instance *ri, struct pt_regs *regs);
+
+regs is as described for the user's uprobe handler.  ri points to
+the uretprobe_instance object associated with the particular function
+instance that is currently returning.  The following fields in that
+object may be of interest:
+- ret_addr: the return address
+- rp: points to the corresponding uretprobe object
+
+In ptrace.h, the regs_return_value(regs) macro provides a simple
+abstraction to extract the return value from the appropriate register
+as defined by the architecture's ABI.
+
+4.3 unregister_*probe
 
 #include <linux/uprobes.h>
 void unregister_uprobe(struct uprobe *u);
+void unregister_uretprobe(struct uretprobe *rp);
 
-Removes the specified probe.  unregister_uprobe() can be called
+Removes the specified probe.  The unregister function can be called
 at any time after the probe has been registered.
 
 5. Uprobes Features and Limitations
@@ -157,12 +214,13 @@ The user is expected to assign values on
 of struct uprobe: pid, vaddr, and handler.  Other members are reserved
 for Uprobes' use.  Uprobes may produce unexpected results if you:
 - assign non-zero values to reserved members of struct uprobe;
-- change the contents of a uprobe object while it is registered; or
-- attempt to register a uprobe that is already registered.
-
-Uprobes allows any number of probes at a particular address.  For a
-particular probepoint, handlers are run in the order in which they
-were registered.
+- change the contents of a uprobe or uretprobe object while it is
+registered; or
+- attempt to register a uprobe or uretprobe that is already registered.
+
+Uprobes allows any number of probes (uprobes and/or uretprobes)
+at a particular address.  For a particular probepoint, handlers are
+run in the order in which they were registered.
 
 Any number of kernel modules may probe a particular process
 simultaneously, and a particular module may probe any number of
@@ -172,8 +230,8 @@ Probes are shared by all threads in a pr
 threads).
 
 If a probed process exits or execs, Uprobes automatically unregisters
-all uprobes associated with that process.  Subsequent attempts to
-unregister these probes will be treated as no-ops.
+all uprobes and uretprobes associated with that process.  Subsequent
+attempts to unregister these probes will be treated as no-ops.
 
 On the other hand, if a probed memory area is removed from the
 process's virtual memory map (e.g., via dlclose(3) or munmap(2)),
@@ -206,6 +264,16 @@ to install a bug fix or to inject faults
 course, has no way to distinguish the deliberately injected faults
 from the accidental ones.  Don't drink and probe.
 
+Since a return probe is implemented by replacing the return
+address with the trampoline's address, stack backtraces and calls
+to __builtin_return_address() will typically yield the trampoline's
+address instead of the real return address for uretprobed functions.
+
+If the number of times a function is called does not match the
+number of times it returns (e.g., if a function exits via longjmp()),
+registering a return probe on that function may produce undesirable
+results.
+
 When you register the first probe at probepoint or unregister the
 last probe probe at a probepoint, Uprobes asks Utrace to "quiesce"
 the probed process so that Uprobes can insert or remove the breakpoint
@@ -227,14 +295,14 @@ Uprobes is intended to interoperate usef
 Documentation/kprobes.txt).  For example, an instrumentation module
 can make calls to both the Kprobes API and the Uprobes API.
 
-A uprobe handler can register or unregister kprobes, jprobes,
-and kretprobes.  On the other hand, a kprobe, jprobe, or kretprobe
-handler must not sleep, and therefore cannot register or unregister
-any of these types of probes.  (Ideas for removing this restriction
-are welcome.)
+A uprobe or uretprobe handler can register or unregister kprobes,
+jprobes, and kretprobes.  On the other hand, a kprobe, jprobe, or
+kretprobe handler must not sleep, and therefore cannot register or
+unregister any of these types of probes.  (Ideas for removing this
+restriction are welcome.)
 
-Note that the overhead of a uprobe hit is several times that of a
-kprobe hit.
+Note that the overhead of a u[ret]probe hit is several times that of
+a kprobe hit.
 
 7. Interoperation with Utrace
 
@@ -281,7 +349,7 @@ be probed.
 - In your report_quiesce callback, register the desired probes.
 (Note that you cannot use the same probe object for both parent
 and child.  If you want to duplicate the probepoints, you must
-create a new set of uprobe objects.)
+create a new set of u[ret]probe objects.)
 
 8. Probe Overhead
 
@@ -290,11 +358,15 @@ On a typical CPU in use in 2007, a uprob
 microseconds to process.  Specifically, a benchmark that hits the same
 probepoint repeatedly, firing a simple handler each time, reports
 250,000 to 300,000 hits per second, depending on the architecture.
+A return-probe hit typically takes 50% longer than a uprobe hit.
+When you have a return probe set on a function, adding a uprobe at
+the entry to that function adds essentially no overhead.
 
 Here are sample overhead figures (in usec) for different architectures.
+u = uprobe; r = return probe; ur = uprobe + return probe
 
 i386: Intel Pentium M, 1495 MHz, 2957.31 bogomips
-3.4 usec
+u = 3.4 usec; r = 4.7 usec; ur = 4.7 usec
 
 x86_64: AMD Opteron 246, 1994 MHz, 3971.48 bogomips
 // TODO
@@ -422,3 +494,122 @@ is called.  To turn off probing, remove 
 
 In /var/log/messages and on the console, you will see a message of the
 form "Probepoint was hit 5 times".
+
+12. Uretprobes Example
+
+Here's a sample kernel module showing the use of a return probe to
+report a function's return values.
+----- cut here -----
+/* uretprobe_example.c */
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/uprobes.h>
+#include <linux/ptrace.h>
+
+/*
+ * Usage:
+ * insmod uretprobe_example.ko pid=<pid> func=<addr> [verbose=0]
+ * where <pid> identifies the probed process, and <addr> is the virtual
+ * address of the probed function.
+ */
+
+static int pid = 0;
+module_param(pid, int, 0);
+MODULE_PARM_DESC(pid, "pid");
+
+static int verbose = 1;
+module_param(verbose, int, 0);
+MODULE_PARM_DESC(verbose, "verbose");
+
+static long func = 0;
+module_param(func, long, 0);
+MODULE_PARM_DESC(func, "func");
+
+static int ncall, nret;
+static struct uprobe usp;
+static struct uretprobe rp;
+
+static void uprobe_handler(struct uprobe *u, struct pt_regs *regs)
+{
+	ncall++;
+	if (verbose)
+		printk(KERN_INFO "Function at %#lx called\n", u->vaddr);
+}
+
+static void uretprobe_handler(struct uretprobe_instance *ri,
+	struct pt_regs *regs)
+{
+	nret++;
+	if (verbose)
+		printk(KERN_INFO "Function at %#lx returns %#lx\n",
+			ri->rp->u.vaddr, regs_return_value(regs));
+}
+
+int init_module(void)
+{
+	int ret;
+
+	/* Register the entry probe. */
+	usp.pid = pid;
+	usp.vaddr = func;
+	usp.handler = uprobe_handler;
+	printk(KERN_INFO "Registering uprobe on pid %d, vaddr %#lx\n",
+		usp.pid, usp.vaddr);
+	ret = register_uprobe(&usp);
+	if (ret != 0) {
+		printk(KERN_ERR "register_uprobe() failed, returned %d\n", ret);
+		return -1;
+	}
+
+	/* Register the return probe. */
+	rp.u.pid = pid;
+	rp.u.vaddr = func;
+	rp.handler = uretprobe_handler;
+	printk(KERN_INFO "Registering return probe on pid %d, vaddr %#lx\n",
+		rp.u.pid, rp.u.vaddr);
+	ret = register_uretprobe(&rp);
+	if (ret != 0) {
+		printk(KERN_ERR "register_uretprobe() failed, returned %d\n",
+			ret);
+		unregister_uprobe(&usp);
+		return -1;
+	}
+	return 0;
+}
+
+void cleanup_module(void)
+{
+	printk(KERN_INFO "Unregistering probes on pid %d, vaddr %#lx\n",
+		usp.pid, usp.vaddr);
+	printk(KERN_INFO "%d calls, %d returns\n", ncall, nret);
+	unregister_uprobe(&usp);
+	unregister_uretprobe(&rp);
+}
+MODULE_LICENSE("GPL");
+----- cut here -----
+
+Build the kernel module as shown in the above uprobe example.
+
+$ nm -p myprog | awk '$3=="myfunc"'
+080484a8 T myfunc
+$ ./myprog &
+$ ps
+  PID TTY          TIME CMD
+ 4367 pts/3    00:00:00 bash
+ 9156 pts/3    00:00:00 myprog
+ 9157 pts/3    00:00:00 ps
+$ su -
+...
+# insmod uretprobe_example.ko pid=9156 func=0x080484a8
+
+In /var/log/messages and on the console, you will see messages such
+as the following:
+kernel: Function at 0x80484a8 called
+kernel: Function at 0x80484a8 returns 0x3
+To turn off probing, remove the module:
+
+# rmmod uretprobe_example
+
+In /var/log/messages and on the console, you will see a message of the
+form "73 calls, 73 returns".
diff -puN arch/i386/kernel/uprobes.c~3-uretprobes arch/i386/kernel/uprobes.c
--- linux-2.6.21-rc6/arch/i386/kernel/uprobes.c~3-uretprobes	2007-04-23 15:30:29.000000000 -0700
+++ linux-2.6.21-rc6-jimk/arch/i386/kernel/uprobes.c	2007-04-23 15:33:12.000000000 -0700
@@ -137,3 +137,43 @@ void uprobe_resume_execution(struct upro
 	else
 		regs->eip = orig_eip + (regs->eip - copy_eip);
 }
+
+/*
+ * Replace the return address with the trampoline address.  Returns
+ * the original return address.
+ */
+unsigned long arch_hijack_uret_addr(unsigned long trampoline_address,
+		struct pt_regs *regs)
+{
+	int nleft;
+	unsigned long orig_ret_addr;
+#define RASIZE (sizeof(unsigned long))
+
+	nleft = copy_from_user(&orig_ret_addr,
+		       (const void __user *)regs->esp, RASIZE);
+	if (unlikely(nleft != 0))
+		return 0;
+
+	if (orig_ret_addr == trampoline_address)
+		/*
+		 * There's another uretprobe on this function, and it was
+		 * processed first, so the return address has already
+		 * been hijacked.
+		 */
+		return orig_ret_addr;
+
+	nleft = copy_to_user((void __user *)regs->esp,
+		       &trampoline_address, RASIZE);
+	if (unlikely(nleft != 0)) {
+		if (nleft != RASIZE) {
+			printk(KERN_ERR "uretprobe_entry_handler: "
+					"return address partially clobbered -- "
+					"pid=%d, %%esp=%#lx, %%eip=%#lx\n",
+					current->pid, regs->esp, regs->eip);
+			BUG();
+		} /* else nothing written, so no harm */
+		return 0;
+	}
+	return orig_ret_addr;
+}
+
diff -puN include/asm-i386/uprobes.h~3-uretprobes include/asm-i386/uprobes.h
--- linux-2.6.21-rc6/include/asm-i386/uprobes.h~3-uretprobes	2007-04-23 15:30:29.000000000 -0700
+++ linux-2.6.21-rc6-jimk/include/asm-i386/uprobes.h	2007-04-23 15:33:12.000000000 -0700
@@ -42,7 +42,7 @@ static inline int arch_validate_probed_i
 	return 0;
 }
 
-/* On i386, the int3 traps leaves eip pointing past the int3 instruction. */
+/* On i386, the int3 traps leaves eip pointing past the int instruction. */
 static inline unsigned long arch_get_probept(struct pt_regs *regs)
 {
 	return (unsigned long) (regs->eip - BP_INSN_SIZE);
@@ -53,4 +53,14 @@ static inline void arch_reset_ip_for_sst
 	regs->eip -= BP_INSN_SIZE;
 }
 
+#define ARCH_SUPPORTS_URETPROBES 1
+static inline void arch_restore_uret_addr(unsigned long ret_addr,
+		struct pt_regs *regs)
+{
+	regs->eip = ret_addr;
+}
+
+extern unsigned long arch_hijack_uret_addr(unsigned long trampoline_addr,
+		struct pt_regs *regs);
+
 #endif				/* _ASM_UPROBES_H */
diff -puN include/linux/uprobes.h~3-uretprobes include/linux/uprobes.h
--- linux-2.6.21-rc6/include/linux/uprobes.h~3-uretprobes	2007-04-23 15:30:29.000000000 -0700
+++ linux-2.6.21-rc6-jimk/include/linux/uprobes.h	2007-04-23 15:33:12.000000000 -0700
@@ -71,6 +71,14 @@ struct uprobe {
 	pid_t tgid;
 };
 
+struct uretprobe_instance;
+typedef void (*uretprobe_handler_t)(struct uretprobe_instance*,struct pt_regs*);
+
+struct uretprobe {
+	struct uprobe u;
+	uretprobe_handler_t handler;
+};
+
 #ifdef CONFIG_UPROBES
 #include <asm/uprobes.h>
 
@@ -87,6 +95,7 @@ enum uprobe_task_state {
 	UPTASK_SLEEPING,	// used when task may not be able to quiesce
 	UPTASK_RUNNING,
 	UPTASK_BP_HIT,
+	UPTASK_TRAMPOLINE_HIT,
 	UPTASK_SSTEP_AFTER_BP
 };
 
@@ -101,7 +110,8 @@ enum uprobe_task_state {
 enum uprobe_ssol_slot_state {
 	SSOL_FREE,
 	SSOL_ASSIGNED,
-	SSOL_BEING_STOLEN
+	SSOL_BEING_STOLEN,
+	SSOL_RESERVED		// e.g., for uretprobe trampoline
 };
 
 /*
@@ -110,7 +120,7 @@ enum uprobe_ssol_slot_state {
  */
 struct uprobe_ssol_slot {
 	/* The slot in the SSOL area that holds the instruction-copy */
-	__user uprobe_opcode_t	*insn;
+	__user uprobe_opcode_t *insn;
 
 	enum uprobe_ssol_slot_state state;
 
@@ -203,12 +213,16 @@ struct uprobe_process {
 	 * We won't free the uprobe_process while...
 	 * - any register/unregister operations on it are in progress; or
 	 * - uprobe_table[] is not empty; or
-	 * - any tasks are SLEEPING in the waitq.
+	 * - any tasks are SLEEPING in the waitq; or
+	 * - any uretprobe_instances are outstanding.
 	 * refcount reflects this.  We do NOT ref-count tasks (threads),
 	 * since once the last thread has exited, the rest is academic.
 	 */
 	struct kref refcount;
 
+	/* Return-probed functions return via this trampoline. */
+	__user uprobe_opcode_t *uretprobe_trampoline_addr;
+
 	/*
 	 * Manages slots for instruction-copies to be single-stepped
 	 * out of line.
@@ -314,9 +328,18 @@ struct uprobe_task {
 	/* Set before running handlers; cleared after single-stepping. */
 	struct uprobe_kimg *active_probe;
 
+	/* LIFO -- active instances */
+	struct hlist_head uretprobe_instances;
+
 	struct mutex mutex;
 };
 
+struct uretprobe_instance {
+	struct uretprobe *rp;
+	unsigned long ret_addr;
+	struct hlist_node hlist;
+};
+
 int register_uprobe(struct uprobe *u);
 void unregister_uprobe(struct uprobe *u);
 
@@ -338,4 +361,18 @@ static inline void unregister_uprobe(str
 {
 }
 #endif	/* CONFIG_UPROBES */
+
+#if defined(CONFIG_UPROBES) && defined(ARCH_SUPPORTS_URETPROBES)
+extern int register_uretprobe(struct uretprobe *rp);
+extern void unregister_uretprobe(struct uretprobe *rp);
+#else
+static inline int register_uretprobe(struct uretprobe *u)
+{
+	return -ENOSYS;
+}
+static inline void unregister_uretprobe(struct uretprobe *u)
+{
+}
+#endif
+
 #endif	/* _LINUX_UPROBES_H */
diff -puN kernel/uprobes.c~3-uretprobes kernel/uprobes.c
--- linux-2.6.21-rc6/kernel/uprobes.c~3-uretprobes	2007-04-23 15:30:29.000000000 -0700
+++ linux-2.6.21-rc6-jimk/kernel/uprobes.c	2007-04-23 15:34:08.000000000 -0700
@@ -40,6 +40,18 @@
 extern int access_process_vm(struct task_struct *tsk, unsigned long addr,
 	void *buf, int len, int write);
 
+static void uretprobe_handle_entry(struct uprobe *u, struct pt_regs *regs,
+	struct uprobe_task *utask);
+static void uretprobe_handle_return(struct pt_regs *regs,
+	struct uprobe_task *utask);
+static void uretprobe_set_trampoline(struct uprobe_process *uproc);
+
+typedef void (*uprobe_handler_t)(struct uprobe*, struct pt_regs*);
+#define URETPROBE_HANDLE_ENTRY ((uprobe_handler_t)-1L)
+#define is_uretprobe(u) (u->handler == URETPROBE_HANDLE_ENTRY)
+/* Point utask->active_probe at this while running uretprobe handler. */
+static struct uprobe_kimg uretprobe_trampoline_dummy_probe;
+
 /*
  * Locking hierarchy:
  * uproc_mutex
@@ -357,6 +369,23 @@ static int quiesce_all_threads(struct up
 	return survivors;
 }
 
+/* Called with utask->mutex and utask->uproc->mutex held. */
+static void uprobe_free_task(struct uprobe_task *utask)
+{
+	struct uretprobe_instance *ri;
+	struct hlist_node *r1, *r2;
+
+	list_del(&utask->list);
+	hlist_for_each_entry_safe(ri, r1, r2, &utask->uretprobe_instances,
+			hlist) {
+		hlist_del(&ri->hlist);
+		kfree(ri);
+		uprobe_put_process(utask->uproc);
+	}
+	mutex_unlock(&utask->mutex);
+	kfree(utask);
+}
+
 /* Runs with uproc_mutex and uproc->mutex held. */
 static void uprobe_free_process(struct uprobe_process *uproc)
 {
@@ -375,8 +404,7 @@ static void uprobe_free_process(struct u
 		 */
 		if (utask->engine)
 			utrace_detach(utask->tsk, utask->engine);
-		mutex_unlock(&utask->mutex);
-		kfree(utask);
+		uprobe_free_task(utask);
 	}
 	if (area->slots)
 		kfree(area->slots);
@@ -423,6 +451,7 @@ static struct uprobe_task *uprobe_add_ta
 	utask->quiescing = 0;
 	utask->uproc = uproc;
 	utask->active_probe = NULL;
+	INIT_HLIST_HEAD(&utask->uretprobe_instances);
 
 	engine = utrace_attach(t, UTRACE_ATTACH_CREATE, p_uprobe_utrace_ops,
 		utask);
@@ -507,6 +536,7 @@ static struct uprobe_process *uprobe_mk_
 	uproc->n_quiescent_threads = 0;
 	INIT_HLIST_NODE(&uproc->hlist);
 	uproc->tgid = p->tgid;
+	uproc->uretprobe_trampoline_addr = NULL;
 
 	uproc->ssol_area.state = SSOL_NOT_SETUP;
 	mutex_init(&uproc->ssol_area.setup_mutex);
@@ -704,6 +734,12 @@ int register_uprobe(struct uprobe *u)
 		uproc_is_new = 1;
 	}
 
+	if (is_uretprobe(u) && IS_ERR(uproc->uretprobe_trampoline_addr)) {
+		/* Previously failed to set up trampoline. */
+		ret = -ENOMEM;
+		goto fail_uproc;
+	}
+
 	INIT_LIST_HEAD(&u->list);
 
 	/* See if we already have a uprobe at the vaddr. */
@@ -889,11 +925,15 @@ static int uprobe_setup_ssol_vma(void)
  * Must be run by a thread in the probed process.
  */
 static enum uprobe_ssol_area_state
-			uprobe_init_ssol(struct uprobe_ssol_area *area)
+			uprobe_init_ssol(struct uprobe_process *uproc)
 {
+	struct uprobe_ssol_area *area = &uproc->ssol_area;
 	struct uprobe_ssol_slot *slot;
 	int i;
 
+	/* Trampoline setup will either fail or succeed here. */
+	uproc->uretprobe_trampoline_addr = ERR_PTR(-ENOMEM);
+
 	 /*
 	  * If we previously probed this process and then removed all
 	  * probes, the vma is still available to us.
@@ -926,6 +966,7 @@ static enum uprobe_ssol_area_state
 			((unsigned long)area->insn_area
 			+ (i * MAX_UINSN_BYTES));
 	}
+	uretprobe_set_trampoline(uproc);
 	return SSOL_SETUP_OK;
 }
 
@@ -944,14 +985,18 @@ static enum uprobe_ssol_area_state
 	mutex_lock(&area->setup_mutex);
 	if (likely(area->state == SSOL_NOT_SETUP))
 		/* Nobody snuck in and set things up ahead of us. */
-		area->state = uprobe_init_ssol(area);
+		area->state = uprobe_init_ssol(uproc);
 	mutex_unlock(&area->setup_mutex);
 	return area->state;
 }
 
 static inline int advance_slot(int slot, struct uprobe_ssol_area *area)
 {
-	return (slot + 1) % area->nslots;
+	/* Slot 0 is reserved for uretprobe trampoline. */
+	slot++;
+	if (slot >= area->nslots)
+		slot = 1;
+	return slot;
 }
 
 /*
@@ -1216,6 +1261,7 @@ static u32 uprobe_report_signal(struct u
 	struct uprobe *u;
 	u32 ret;
 	unsigned long probept;
+	int hit_uretprobe_trampoline = 0;
 
 	utask = rcu_dereference((struct uprobe_task *)engine->data);
 	BUG_ON(!utask);
@@ -1228,9 +1274,15 @@ static u32 uprobe_report_signal(struct u
 	 * here because we have to do it before handling the first
 	 * probepoint hit, the probed process has to do it, and this may
 	 * be the first time our probed process runs uprobes code.
+	 *
+	 * We need the SSOL area for the uretprobe trampoline even if
+	 * this architectures doesn't single-step out of line.
 	 */
 	uproc = utask->uproc;
-	if (uproc->sstep_out_of_line &&
+	if (
+#ifndef ARCH_SUPPORTS_URETPROBES
+		uproc->sstep_out_of_line &&
+#endif
 			unlikely(uprobe_verify_ssol(uproc) != SSOL_SETUP_OK))
 		uproc->sstep_out_of_line = 0;
 
@@ -1238,6 +1290,16 @@ static u32 uprobe_report_signal(struct u
 	switch (utask->state) {
 	case UPTASK_RUNNING:
 		probept = arch_get_probept(regs);
+		hit_uretprobe_trampoline = (probept == (unsigned long)
+			uproc->uretprobe_trampoline_addr);
+		if (hit_uretprobe_trampoline) {
+			uprobe_get_process(uproc);
+			utask->state = UPTASK_TRAMPOLINE_HIT;
+			utask->active_probe = &uretprobe_trampoline_dummy_probe;
+			uretprobe_handle_return(regs, utask);
+			goto bp_done;
+		}
+
 		down_read(&uproc->utable_rwsem);
 		uk = find_uprobe(uproc, probept, 1);
 		up_read(&uproc->utable_rwsem);
@@ -1250,7 +1312,9 @@ static u32 uprobe_report_signal(struct u
 
 		if (likely(uk->state == UPROBE_BP_SET)) {
 			list_for_each_entry(u, &uk->uprobe_list, list) {
-				if (u->handler)
+				if (is_uretprobe(u))
+					uretprobe_handle_entry(u, regs, utask);
+				else if (u->handler)
 					u->handler(u, regs);
 			}
 		}
@@ -1277,6 +1341,8 @@ static u32 uprobe_report_signal(struct u
 		else
 			uprobe_post_ssin(utask, uk);
 
+bp_done:
+		/* Note: Can come here after running uretprobe handlers */
 		utask->active_probe = NULL;
 		ret = UTRACE_ACTION_HIDE | UTRACE_SIGNAL_IGN
 			| UTRACE_ACTION_NEWSTATE;
@@ -1288,6 +1354,13 @@ static u32 uprobe_report_signal(struct u
 		} else
 			mutex_unlock(&utask->mutex);
 
+		if (hit_uretprobe_trampoline)
+			/*
+			 * It's possible that the uretprobe_instance
+			 * we just recycled was the last reason for
+			 * keeping uproc around.
+			 */
+			uprobe_put_process(uproc);
 		break;
 	default:
 		mutex_unlock(&utask->mutex);
@@ -1665,3 +1738,166 @@ __initcall(init_uprobes);
 
 EXPORT_SYMBOL_GPL(register_uprobe);
 EXPORT_SYMBOL_GPL(unregister_uprobe);
+
+#ifdef ARCH_SUPPORTS_URETPROBES
+
+/* Called when the entry-point probe u is hit. */
+static void uretprobe_handle_entry(struct uprobe *u, struct pt_regs *regs,
+	struct uprobe_task *utask)
+{
+	struct uretprobe_instance *ri;
+	unsigned long trampoline_addr;
+
+	if (IS_ERR(utask->uproc->uretprobe_trampoline_addr))
+		return;
+	trampoline_addr = (unsigned long)
+		utask->uproc->uretprobe_trampoline_addr;
+	ri = (struct uretprobe_instance *)
+		kmalloc(sizeof(struct uretprobe_instance), GFP_USER);
+	if (!ri)
+		return;
+	ri->ret_addr = arch_hijack_uret_addr(trampoline_addr, regs);
+	if (likely(ri->ret_addr)) {
+		ri->rp = container_of(u, struct uretprobe, u);
+		INIT_HLIST_NODE(&ri->hlist);
+		hlist_add_head(&ri->hlist, &utask->uretprobe_instances);
+		/* We ref-count outstanding uretprobe_instances. */
+		uprobe_get_process(utask->uproc);
+	} else
+		kfree(ri);
+}
+
+/*
+ * For each uretprobe_instance pushed onto the LIFO for the function
+ * instance that's now returning, call the handler, free the ri, and
+ * decrement the uproc's ref count.  Caller ref-counts uproc, so we
+ * should never hit zero in this function.
+ *
+ * Returns the original return address.
+ */
+static unsigned long uretprobe_run_handlers(struct uprobe_task *utask,
+		struct pt_regs *regs, unsigned long trampoline_addr)
+{
+	unsigned long ret_addr;
+	struct hlist_head *head = &utask->uretprobe_instances;
+	struct uretprobe_instance *ri;
+	struct hlist_node *r1, *r2;
+
+	hlist_for_each_entry_safe(ri, r1, r2, head, hlist) {
+		if (ri->rp && ri->rp->handler)
+			ri->rp->handler(ri, regs);
+		ret_addr = ri->ret_addr;
+		hlist_del(&ri->hlist);
+		kfree(ri);
+		uprobe_put_process(utask->uproc);
+		if (ret_addr != trampoline_addr)
+			/*
+			 * This is the first ri (chronologically) pushed for
+			 * this particular instance of the probed function.
+			 */
+			return ret_addr;
+	}
+	BUG();
+	return 0;
+}
+
+/* Called when the uretprobe trampoline is hit. */
+static void uretprobe_handle_return(struct pt_regs *regs,
+	struct uprobe_task *utask)
+{
+	unsigned long orig_ret_addr;
+	orig_ret_addr = uretprobe_run_handlers(utask, regs,
+		(unsigned long) utask->uproc->uretprobe_trampoline_addr);
+	arch_restore_uret_addr(orig_ret_addr, regs);
+}
+
+int register_uretprobe(struct uretprobe *rp)
+{
+	if (!rp || !rp->handler)
+		return -EINVAL;
+	rp->u.handler = URETPROBE_HANDLE_ENTRY;
+	return register_uprobe(&rp->u);
+}
+
+/*
+ * rp has just been unregistered.  Its uretprobe_instances have to hang
+ * around 'til their associated instances return.  Zap ri->rp for each
+ * one to indicate unregistration.
+ *
+ * Called and returns with no locks held.
+ */
+static void zap_uretprobe_instances(struct uretprobe *rp,
+	struct uprobe_process *uproc)
+{
+	struct uprobe_task *utask;
+
+	if (!uproc)
+		return;
+
+	mutex_lock(&uproc->mutex);
+	list_for_each_entry(utask, &uproc->thread_list, list) {
+		struct hlist_node *r;
+		struct uretprobe_instance *ri;
+
+		mutex_lock(&utask->mutex);
+		hlist_for_each_entry(ri, r, &utask->uretprobe_instances, hlist)
+			if (ri->rp == rp)
+				ri->rp = NULL;
+		mutex_unlock(&utask->mutex);
+	}
+	mutex_unlock(&uproc->mutex);
+}
+
+void unregister_uretprobe(struct uretprobe *rp)
+{
+	struct uprobe_process *uproc = NULL;
+	if (!rp)
+		return;
+	if (rp->u.uk)
+		uproc = rp->u.uk->uproc;
+	unregister_uprobe(&rp->u);
+	if (rp->u.status == 0)
+		zap_uretprobe_instances(rp, uproc);
+}
+
+/*
+ * uproc->ssol_area has been successfully set up.  Establish the
+ * uretprobe trampoline in slot 0.
+ */
+static void uretprobe_set_trampoline(struct uprobe_process *uproc)
+{
+	uprobe_opcode_t bp_insn = BREAKPOINT_INSTRUCTION;
+	struct uprobe_ssol_area *area = &uproc->ssol_area;
+	struct uprobe_ssol_slot *slot = &area->slots[0];
+
+	if (access_process_vm(current, (unsigned long) slot->insn,
+			&bp_insn, BP_INSN_SIZE, 1) == BP_INSN_SIZE) {
+		uproc->uretprobe_trampoline_addr = slot->insn;
+		slot->state = SSOL_RESERVED;
+		area->next_slot = 1;
+	} else {
+		printk(KERN_ERR "uretprobes disabled for pid %d:"
+			" cannot set uretprobe trampoline at %p\n",
+			uproc->tgid, slot->insn);
+	}
+}
+
+EXPORT_SYMBOL_GPL(register_uretprobe);
+EXPORT_SYMBOL_GPL(unregister_uretprobe);
+
+#else	/* ! ARCH_SUPPORTS_URETPROBES */
+
+static void uretprobe_handle_entry(struct uprobe *u, struct pt_regs *regs,
+	struct uprobe_task *utask)
+{
+}
+static void uretprobe_handle_return(struct pt_regs *regs,
+	struct uprobe_task *utask)
+{
+}
+static void uretprobe_set_trampoline(struct uprobe_process *uproc)
+{
+}
+
+#endif /* ARCH_SUPPORTS_URETPROBES */
+
_