This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Script to look for system call errors
- From: Eugene Teo <eugene at redhat dot com>
- To: systemtap at sources dot redhat dot com
- Date: Wed, 01 Apr 2009 22:18:16 +0800
- Subject: Script to look for system call errors
Hi all,
Last night I was working on a script to look for system call errors.
This is particularly useful for identifying misbehaving userspace
applications. The script prints out useful information about the top 20
errors every 5 secs.
With the script, I found out that mixer_applet2 wakes up very
frequently[1], gnome-panel tries to add a collection of watches every
second for files that do not exist, hald tries to open some
battery-related sysfs files over and over again even though I am running
the script in a virtual machine. mjw has a couple of interesting
observations with the script as well.
Take a look at the example output in errsnoop.txt.
Any feedback on this script would be appreciated.
[1] http://bugzilla.gnome.org/show_bug.cgi?id=370937
Thanks, Eugene
#!/bin/sh
//usr/bin/env stap -DSTP_NO_OVERLOAD $@ $0; exit $?
# errsnoop.stp
# Copyright (C) 2009 Eugene Teo <eugeneteo@kernel.sg>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License version 2 as
# published by the Free Software Foundation.
#
# attack "stupid userspace" apps
#
global error, trace
probe syscall.* {
trace[tid(), probefunc()] = argstr
}
probe syscall.*.return {
t = tid(); p = probefunc()
if ([t, p] in trace) {
argstr = trace[t, p]
delete trace[t, p]
}
errstr = errno_str(errno = returnval())
if (errno < 0 && strlen(errstr) > 0) {
errstr = sprintf("%3d (%s)", -errno, errstr)
error[probefunc(), execname(), pid(), errstr, argstr] <<< 1
}
}
probe timer.s(5) {
printf("\033[2J\033[1;1H")
printf("%22s %16s %5s %5s %-12s %s\n",
"SYSCALL", "PROCESS", "PID", "COUNT", "ERRSTR", "ARGSTR")
foreach([func, proc, pid, errstr, argstr] in error- limit 20) {
printf("%22s %16s %5d %5d %-12s %s\n", func, proc, pid,
@count(error[func, proc, pid, errstr, argstr]),
errstr, argstr)
}
delete error
}
SYSCALL PROCESS PID COUNT ERRSTR ARGSTR
sys_read mixer_applet2 2641 77 11 (EAGAIN) 3, 0x08d9d834, 4096
sys_read gnome-settings- 2484 3 11 (EAGAIN) 3, 0x0931b604, 4096
sys_read metacity 2485 3 11 (EAGAIN) 3, 0x0816cdd4, 4096
sys_read gnome-screensav 2541 3 11 (EAGAIN) 3, 0x09dee5d4, 4096
sys_inotify_add_watch gnome-panel 2488 2 2 (ENOENT) 18, "/home/test/.config/menus", 16789454
sys_read gnome-power-man 2585 1 11 (EAGAIN) 3, 0x084c3444, 4096
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/home/test/.config/menus/applications-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/home/test/.config/menus/settings-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/home/test/.config/menus/preferences-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/etc/xdg/menus/system-settings-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/home/test/.config/menus/system-settings-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/etc/xdg/menus/server-settings-merged", 16789454
sys_inotify_add_watch gnome-panel 2488 1 2 (ENOENT) 18, "/home/test/.config/menus/server-settings-merged", 16789454
sys_read gnome-panel 2488 1 11 (EAGAIN) 3, 0x081e435c, 4096
sys_read Xorg 2002 1 11 (EAGAIN) 30, 0x0a0f7ea8, 4096
sys_wait4 mono 2639 1 10 (ECHILD) 2639, 0x002972c4, WNOHANG, 0x00000000
sys_inotify_add_watch gnome-screensav 2541 1 2 (ENOENT) 16, "/home/test/.config/menus", 16789454
sys_inotify_add_watch gnome-screensav 2541 1 2 (ENOENT) 16, "/etc/xdg/menus/gnome-screensavers-merged", 16789454
sys_inotify_add_watch gnome-screensav 2541 1 2 (ENOENT) 16, "/home/test/.config/menus/gnome-screensavers-merged", 16789454
^C