Chapter 4. User-space Probing

4.1. User-Space Events
4.2. Accessing User-Space Target Variables
4.3. User-Space Stack Backtraces
SystemTap initially focused on kernel-space probing. Because there are many instances where user-space probing can help diagnose a problem, SystemTap 0.6 added support to allow probing user-space processes. SystemTap can probe the entry into and return from a function in user-space processes, probe predefined markers in user-space code, and monitor user-process events.
SystemTap requires the uprobes module to perform user-space probing. If your Linux kernel is version 3.5 or higher, it already includes uprobes. To verify that the current kernel supports uprobes natively, run the following command:
grep CONFIG_UPROBES /boot/config-`uname -r`
If uprobes is integrated, the output of this command is as follows:
CONFIG_UPROBES=y
If you are running a kernel prior to version 3.5, SystemTap automatically builds the uprobes module. However, you also need the utrace kernel extensions required by the SystemTap user-space probing to track various user-space events. More details about the utrace infrastructure are available at http://sourceware.org/systemtap/wiki/utrace. To determine whether the currently running Linux kernel provides the needed utrace support, type the following at a shell prompt:
grep CONFIG_UTRACE /boot/config-`uname -r`
If the Linux kernel supports user-space probing, the command produces the following output:
CONFIG_UTRACE=y

4.1. User-Space Events

All user-space event probes begin with process. You can limit the process events to a specific running process by specifying the process ID. You can also limit the process events to monitor a particular executable by specifying the path to the executable (PATH). SystemTap makes use of the PATH environment variable, which allows you to use both the name used on the command-line to start the executable and the absolute path to the executable.
Several of the user-space probe events limit their scope to a particular executable name (PATH), because SystemTap must use debug information to statically analyze where to place the probes. But for many user-space probe events, the process ID and executable name are optional. Any process event in the list below that include process ID or the path to the executable must include those arguments. The process ID and path to the executable are optional for the process events that do not list them:
process("PATH").function("function")
The entry to the user-space function function for the executable PATH. This event is the user-space analogue of the kernel.function("function") event. It allows wildcards for the function function and .return suffix.
process("PATH").statement("statement")
The earliest instruction in the code for statement. This is the user-space analogue of kernel.statement("statement").
process("PATH").mark("marker")
The static probe point marker defined in PATH. You can use wildcards for marker to specify multiple marks with a single probe. The static probe points may also have numbered arguments ($1, $2, and so on) available to the probe.
A variety of user-space packages such as Java include these static probe points. Most packages that provide static probe points also provide aliases for the raw user-space mark events. Below is one such alias for the x86_64 Java hotspot JVM:
probe hotspot.gc_begin =
  process("/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so").mark("gc__begin")
process.begin
A user-space process is created. You can limit this to a particular process ID or a full path to the executable.
process.thread.begin
A user-space thread is created. You can limit this to a particular process ID or a full path to the executable.
process.end
A user-space process dies. You can limit this to a particular process ID or a full path to the executable.
process.thread.end
A user-space thread is destroyed. You can limit this to a particular process ID or a full path to the executable.
process.syscall
A user-space process makes a system call. The system call number is available in the $syscall context variable, and the fist six arguments are available in $arg1 through $arg6. The .return suffix places the probe at the return from the system call. For syscall.return, the return value is available through the $return context variable.
You can limit this to a particular process ID or a full path to the executable.