Product SiteDocumentation Site

Chapter 4. User-Space Probing

4.1. User-Space Events
4.2. Accessing User-Space Target Variables
4.3. User-Space Stack Backtraces
SystemTap initially focused on kernel-space probing. However, there are many instances where user-space probing can help diagnose a problem. SystemTap 0.6 added support to allow probing user-space processes. SystemTap includes support for probing the entry into and return from a function in user-space processes, probing predefined markers in user-space code, and monitoring user-process events.
The SystemTap user-space probing requires the utrace kernel extensions which provide an API for tracking various user-space events. More details about the utrace infrastructure are available at The following command determines whether the currently running Linux kernel provides the needed utrace support:
grep CONFIG_UTRACE /boot/config-`uname -r`
If the Linux kernel support user-space probing, the following output is printed:
The SystemTap user-space probing also needs the uprobes kernel module. If the uprobes kernel module is not available, you will see an error message like the following when attempting to run a script that requires the uprobes kernel module:
SystemTap's version of uprobes is out of date.
As root, or a member of the 'root' group, run
"make -C /usr/share/systemtap/runtime/uprobes".
Pass 4: compilation failed.  Try again with another '--vp 0001' option.
If this occurs, you need to generate a uprobes.ko module for the kernel as directed.

4.1. User-Space Events

All user-space event probes begin with process. The process events can be limited to a specific running process by specifying the process ID. The process events can also be limited to monitoring a particular executable by specifying the path to executable (PATH). SystemTap makes use of the PATH environment variable, so both the name used on the command-line to start the executable and the absolute path to the executable can be used. Several of user-space probe events limit their scope to a particular executable name (PATH) because SystemTap must use debug information to statically analyzed where to places the probes, but for many user-space probes events the process ID and executable name are optional. Any process event in the list below that include process ID or the path to the executable must include those arguments. The process ID and path to the executable are optional for the process events that do not list them:
The entry to the user-space function function for the executable PATH. This event is the user-space analogue of the kernel.function("function") event. It allows wildcards for the function function and .return suffix.
The earliest instruction in the code for statement. This is the user-space analogue of kernel.statement("statement").
The static probe point marker defined in PATH. Wildcards can be used for marker to specify mutiple marks with a single probe. The static probe points may also have numbered arguments ($1, $2, etc.) available to the probe. A variety of user-space packages such as Java include these static probe points. Most packages that provide static probe points also provide aliases for the raw user-space mark events. Below is one such alias for the x86_64 Java hotspot JVM:
probe hotspot.gc_begin =
User-space process is created. This can be limited to a particular process ID or a full path to the executable.
User-space thread is created. This can be limited to a particular process ID or a full path to the executable.
User-space process died. This can be limited to a particular process ID or a full path to the executable.
User-space thread is destroyed. This can be limited to a particular process ID or a full path to the executable.
User-space process makes a system call. The system call number is available via $syscall context variable, and the fist six arguments are available via $arg1 through $arg6. The ".return" suffix will place the probe at the return from the system call. For the "syscall.return" the return value is available through the $return context variable. This can be limited to a particular process ID or a full path to the executable.