script from chris mason

[Wenji Huang <wenji dot huang at oracle dot com>]

Hi,

This is the script from Chris Mason <chris.mason@oracle.com>. It could 
find the most common causes of schedule during the AIO io_submit call. 
I think it's very good that a kernel developer can use systemtap and 
share his script.

Please review it and where it is putted?   how about in examples?

Thanks,
Wenji

#!/bin/env stap
#
# Copyright (C) 2007 Oracle Corp.
#
# This was implemented to find the most common causes of schedule during
# the AIO io_submit call.  It does this by recording which pids are inside
# AIO, and recording the current stack trace if one of those pids is
# inside schedule.
# When the probe exits, it prints out the 30 most common call stacks for
# schedule().

global in_iosubmit
global traces
global sorted

/*
 * add a probe to sys_io_submit, on entry, record in the in_iosubmit
 * hash table that this proc is in io_submit
 */
probe kernel.function("sys_io_submit") {
    p = pid()
    in_iosubmit[p] = 1
}

/*
 * when we return from sys_io_submit, record that we're no longer there
 */
probe kernel.function("sys_io_submit").return {
    p = pid()

    /* this assumes a given proc will do lots of io_submit calls, and
     * so doesn't do the more expensive "delete in_iosubmit[p]".  If
     * there are lots of procs doing small number of io_submit calls,
     * the hash may grow pretty big, so using delete may be better
     */
    in_iosubmit[p] = 0
}

/*
 * every time we call schedule, check to see if we started off in
 * io_submit.  If so, record our backtrace into the traces histogram
 */
probe kernel.function("schedule") {
    p = pid()
    in_io = in_iosubmit[p]
    if (in_io != 0) {
	str = backtrace()
	traces[str] += 1

	/*
	 * change this to if (1) if you want a backtrace every time
	 * you go into schedule from io_submit.  Unfortunately, the traces
	 * saved into the traces histogram above are truncated to just a
	 * few lines.  so the only way to see the full trace is via the
	 * more verbose print_backtrace() right here.
	 */
	if (0) {
		printf("schedule in io_submit!\n")
		print_backtrace()
	}
    }
}

/*
 * when stap is done (via ctrl-c) go through the record of all the
 * trace paths and print the 30 most common.
 */
probe end {
	foreach (str in traces) {
		num = traces[str]
		sorted[ num, str] = 1
	}
	foreach ([num-, str] in sorted limit 30) {
		printf("%d:", num)
		print_stack(str);
	}
}

1496: 0xa00000010059af60 : schedule+0x0/0x1a00 []
0xa00000010059e390 : io_schedule+0x50/0x80 []
0xa00000010036d310 : get_request_wait+0x1d0/0x240 []
0xa00000010036f0a0 : __make_request+0x2e0/0xec0 []
0xa00000010036fec0 : generic_make_request+0x240/0x4e0 []
0xa0000001003702d0 : submit_bio+0x170/0x340 []
0xa000000100182760 : dio_bio_submit+0x120/0x140 []