This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Discussion items from the Collab Summit
- From: Elena Zannoni <elena dot zannoni at oracle dot com>
- To: systemtap <systemtap at sources dot redhat dot com>
- Date: Mon, 04 May 2009 11:28:12 -0400
- Subject: Discussion items from the Collab Summit
These are mine and Christoph's notes from the discussions we had at the
Collaboration Summit in SFO, in April. I've added some clarifications,
comments and updates from Roland and Frank as well.
Attendes: Renzo Davoli, Mathieu Desnoyers, Jake Edge, Frank Ch. Eigler,
Christoph Hellwig, Masami Hiramatsu, Jim Keniston, Roland McGrath,
Steven Rostedt, Josh Stone, Elena Zannoni. Plus James Bottomley (day 2)
Keiichiro Tokunaga (from Fujitsu)
Thursday afternoon:
Kernel Tracing items:
- make DEFINE_TRACE work in modules (Steve)
- investigate markers removal (Christoph, Matthew)
- the 25 magic google tracpoints (Matthew)
- make the two major tracepoint implementations interchangeable
(Matthew, Steve) (working on a common ring buffer API)
- get djprobes and the instruction decoder upstream (Masami)
Utrace and userspace probing:
- get arm and mips converted to regsets and uprobes, set a cut off
date for others (Christoph, Roland)
*** Update 4/24/09: arm work finished by Roland. (posted to lkml?)
Christoph pinged the mips maintainer, no reply seen.
- more ptrace cleanups to prepare for utrace (Oleg) [In progress]
- in-kernel gdb server for debugging userspace (Frank)
- get uprobes upstream piecemail, including backing the gdbserver(Jim)
From Friday morning:
- Include Roland in email about kbuild patch for separate debuginfo
sent by Wenji Huang (Elena)
- elfutils debuginfo reduction (duplicates eliminations): ROland
working on it, goal is Fedora 12 timeframe.
- Look up old patch from crashdump people about pulling type info into
a common .h file for debuginfo purposes. (SLES9 or SLES10) (Christoph?)
- Talk to Arnaldo about his CTF work. Redhat folks to have a meeting
with him.
*** Update 4/24/09(Roland): Acme working on a kbuild scheme using his
tools herein the kernel would compile with -g but have DWARF replaced
with CTF in each .o during compilation. This intends to avoid the "slow
compile" problem cited by some kernel hackes, and believed to be slow
link phase and disk i/o. (We'll see how quick it is.)
That would yield CTF for the kernel, which has at least type layout
information and some minimal symbolic info about functions. That is
intended to suffice for function entry probes with parameters, but it
does not handle the parameter ABI problem. (As far as we can tell,
Sun only deals with one presumed function call ABI.)
- James had a comment about how difficult it is for stap/kretprobes to
probe at the end of a function. Seems that epilogue or last instruction
of a function (upon return) is hard to probe. He is interested not just
in the return value, but in the value of the locals before exit. Gcc-4.4
(current fedora and future rhel6 compiler) should produce correct info.
Systemtap PR: 10056.
What James wants is *new* compiler information to identify a site at
the beginning of the epilogue as a good probe site. He believes that
at such a place, the locals (now technically dead) will in fact still
be accessible in stack slots not clobbered before the epilogue. If
that's in fact true for a particular site of interest, then the idea
presupposes that GCC's DWARF location information for that PC is
accurate about whatever happens to be accessible still. AFAIK we
don't especially suspect any problems with that location information.
The DWARF format (line info) includes an epilogue_begin marker feature.
This is what would be used for finding the probe site James wants.
There was past talk (and maybe even some unfinished work) to make GCC
emit those, but GCC 4.4 does NOT emit those markers.
- how good is stap at working on code that has no associated
debuginfo. For markers, Josh has added a patch that can generate
debuginfo on the fly. To be clear, it generates sufficient information
for type layouts.
That is enough to do interesting things with tracepoint parameters.
(This is the more useful half of what we could get from CTF.)
0.9.6 will have two new features for this.
- Revisit which (if any) parts of systemtap code tree could be pushed
into kernel tree.
- Future directions: there was agreement that doing userspace
debugging via .ko's is not the best solution in the long term, but
currently is the only way there is to do userspace probing.
- Complaints about stap shutting itself down (from Mathieu) when doing
irq monitoring.
The germ of good idea to take away from this (beyond documenting
-DINTERRUPTIBLE and some other tweaks already done) is that we may
want to teach systemtap to be a little more tolerant of reentrancy.
For probe handlers that simply trace (without local vars, without many
temporaries), with a reentrant-tolerant printf(), we can probably do
OK -- without losing the events Mathieu was worried about, and without
-DINTERRUPTIBLE=0. Future work, no PR yet.
- Idea was floated around to define a stable api into the kernel for
which stap is guaranteed not to break from one kernel version to the
next. Mathieu proposed to keep a part of the stap self testing module
into the kernel so that kernel developers can test themselves if they
break something. Idea not practical, since it puts the burden on kernel
community to assure that out of tree code keeps working. Problem is that
stap doesn't use a predefined set of probe points. This is the whole
idea behind the current stap design, that is, to get away from the
static approach of dtrace, and to let you probe everything because of
the use of the debuginfo. I.e. "API" is potentially the whole kernel
code. However, the translator-generated and runtime-boilerplate code use
a relatively small number of exported functions as the API. This is one
reason why the runtime etc. is in fact remarkably stable from kernel
version to kernel version. Maybe 1-3 changes per kernel release are
needed. The tapsets are a different matter. Reorienting them with a
preference for tracepoints will improve them too.
WRT keeping some stap tests in the kernel tree, one problem with this
idea is that while the kernel developer can then instantly fix this
hypothetical self-test module, and potentially make the self-test
compile ("closing the feedback loop"), there is a hole in the feedback
to the systemtap developers. Unless this "self-test module" is in fact
complete, we still need to carry the fuller runtime/etc. code in
systemtap, so we'd need to receive & redistribute those patches
- Mathieu mentioned gdb tracepoints: continuous debugging. Similar to
systemtap idea. Collect info at breakpoints and continue the program.
Code Sourcery working on that.
--
Elena Zannoni, Oracle
Senior Engineering Manager, Tools/Languages - Linux Engineering
Blog: http://blogs.oracle.com/ezannoni
Email: elena.zannoni@oracle.com