comments from prospective VM tapset author

[Jim Keniston <jkenisto at us dot ibm dot com>]

Here are notes from conversations with Darren Hart, who is working on VM
instrumentation that we hope will lead to a tapset.  So far, he's happy
with kprobes, and will continue as best he can in C.  As we get more
details about what a SystemTap tapset is supposed to look like, we'll
plan a conversion to a tapset.

Keep in mind that the goal of these conversations was to gather
information from Darren, not to sell him on the benefits of SystemTap.

Jim
--
Summary of conversations with Darren Hart Mon 4/25/05 and Wed 4/27/05

Darren works for IBM's Linux Technology Center.  He has expertise in
the scheduler and, recently, in VM.  He's interested in kprobes and
SystemTap for two main reasons:
1. He's chasing down a VM performance problem, and these tools sound
like they could help.
2. He has the expertise and the willingness to create (or help create)
SystemTap-based instrumentation (AKA a tapset) for the VM subsystem.

Vara and I met with Darren for about 90 minutes on 4/25/05.  Darren and
I also talked for 10-15 minutes 1-2 days later.

Darren has been using kprobes and jprobes, and he really likes them.
So far, access to function args via jprobes has been sufficient for
him, but he foresees the need to access local variables in the middle
of a function.  We talked about how with kprobes, you typically have to
use a disassembler to figure out where in a function to place a
probepoint, and to verify in what register or stack location the
variable resides.

We talked about ways in which SystemTap could provide access to such
variables and code locations more conveniently using debug info.
Darren didn't like the idea of having to switch to a new language in
order to do his instrumentation.  His idea of what the instrumentation
should be includes the following points:

- coded in C

- SystemTap provides kernel functions and/or macros that provide
standardized, convenient access to local variables in a probed
function.

- Instrumentation is a library of functions similar to what Vara
envisioned, where each instrumentation function collects an advertised
set of values (typically only a few, maybe 10 max) and makes them
available through a standard mechanism (to be defined by us SystemTap
folks).  A SystemTap-generated handler would call such a function to
obtain the advertised values.

We discussed the possibility of making the SystemTap runtime available
for use by the kprobes-based instrumentation that Darren is starting to
write.  I was doubtful from what little I remembered about the runtime,
but I said I would investigate.

Darren provided the attached file, vm_instrumentation.txt, which
summarizes some thoughts on what instrumentation SystemTap should
provide in the VM area.  For example, SystemTap should be able to
provide at least the VM info provided by /proc/meminfo, vmstat,
and atsar.  Some of the stats he listed are probably of special
interest to the performance problem that he's analyzing.  Darren
would like to be able to report the values of counters and levels,
function timings, return values, and function args.  (I pointed out
that function timings would have to take into account the overhead
of the kprobes-related traps.)  He would like to be able to record
stats on a per-process and/or per-CPU basis.

TAPSET TEMPLATE
I gave Darren a copy of Frank's "tapset/script information template"
from 4/15/05, and asked him to review it as someone who might be asked
to fill it out.  His initial response was that the typical subsystem
expert would probably prefer to document their instrumentation after
the fact, touching on whatever points seem relevant to him/her.

He gave me his review comments 1-2 days later.  His written comments
included the following:
- "Seems like a lot.  Who fills it out?  Who reads it?"
- Background should be combined with Motivation.
- Target software should be obvious from Motivation/Background.
- Interesting values, Data collection, and Data presentation should be
combined.

He said he could see the value of the items requested, but that not all
items would be relevant for all instrumentation sets.

Attachment: vm_instrumentation.txt
Description: Text document