This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Discussion items from the Collab Summit


These are mine and Christoph's notes from the discussions we had at the Collaboration Summit in SFO, in April. I've added some clarifications, comments and updates from Roland and Frank as well.


Attendes: Renzo Davoli, Mathieu Desnoyers, Jake Edge, Frank Ch. Eigler, Christoph Hellwig, Masami Hiramatsu, Jim Keniston, Roland McGrath, Steven Rostedt, Josh Stone, Elena Zannoni. Plus James Bottomley (day 2)
Keiichiro Tokunaga (from Fujitsu)


Thursday afternoon:

Kernel Tracing items:

  - make DEFINE_TRACE work in modules (Steve)
  - investigate markers removal (Christoph, Matthew)
  - the 25 magic google tracpoints (Matthew)
  - make the two major tracepoint implementations interchangeable
    (Matthew, Steve) (working on a common ring buffer API)

- get djprobes and the instruction decoder upstream (Masami)

Utrace and userspace probing:

- get arm and mips converted to regsets and uprobes, set a cut off date for others (Christoph, Roland)

*** Update 4/24/09: arm work finished by Roland. (posted to lkml?) Christoph pinged the mips maintainer, no reply seen.


- more ptrace cleanups to prepare for utrace (Oleg) [In progress] - in-kernel gdb server for debugging userspace (Frank) - get uprobes upstream piecemail, including backing the gdbserver(Jim)


From Friday morning:


- Include Roland in email about kbuild patch for separate debuginfo sent by Wenji Huang (Elena)

- elfutils debuginfo reduction (duplicates eliminations): ROland working on it, goal is Fedora 12 timeframe.

- Look up old patch from crashdump people about pulling type info into a common .h file for debuginfo purposes. (SLES9 or SLES10) (Christoph?)

- Talk to Arnaldo about his CTF work. Redhat folks to have a meeting with him.

*** Update 4/24/09(Roland): Acme working on a kbuild scheme using his tools herein the kernel would compile with -g but have DWARF replaced
with CTF in each .o during compilation. This intends to avoid the "slow
compile" problem cited by some kernel hackes, and believed to be slow link phase and disk i/o. (We'll see how quick it is.)
That would yield CTF for the kernel, which has at least type layout
information and some minimal symbolic info about functions. That is
intended to suffice for function entry probes with parameters, but it
does not handle the parameter ABI problem. (As far as we can tell,
Sun only deals with one presumed function call ABI.)




- James had a comment about how difficult it is for stap/kretprobes to probe at the end of a function. Seems that epilogue or last instruction of a function (upon return) is hard to probe. He is interested not just in the return value, but in the value of the locals before exit. Gcc-4.4 (current fedora and future rhel6 compiler) should produce correct info.

Systemtap PR: 10056.

What James wants is *new* compiler information to identify a site at
the beginning of the epilogue as a good probe site.  He believes that
at such a place, the locals (now technically dead) will in fact still
be accessible in stack slots not clobbered before the epilogue.  If
that's in fact true for a particular site of interest, then the idea
presupposes that GCC's DWARF location information for that PC is
accurate about whatever happens to be accessible still.  AFAIK we
don't especially suspect any problems with that location information.

The DWARF format (line info) includes an epilogue_begin marker feature.
This is what would be used for finding the probe site James wants.
There was past talk (and maybe even some unfinished work) to make GCC
emit those, but GCC 4.4 does NOT emit those markers.


- how good is stap at working on code that has no associated debuginfo. For markers, Josh has added a patch that can generate debuginfo on the fly. To be clear, it generates sufficient information for type layouts.
That is enough to do interesting things with tracepoint parameters.
(This is the more useful half of what we could get from CTF.)
0.9.6 will have two new features for this.



- Revisit which (if any) parts of systemtap code tree could be pushed into kernel tree.


- Future directions: there was agreement that doing userspace debugging via .ko's is not the best solution in the long term, but currently is the only way there is to do userspace probing.

- Complaints about stap shutting itself down (from Mathieu) when doing irq monitoring.

The germ of good idea to take away from this (beyond documenting
-DINTERRUPTIBLE and some other tweaks already done) is that we may
want to teach systemtap to be a little more tolerant of reentrancy.
For probe handlers that simply trace (without local vars, without many
temporaries), with a reentrant-tolerant printf(), we can probably do
OK -- without losing the events Mathieu was worried about, and without
-DINTERRUPTIBLE=0.  Future work, no PR yet.


- Idea was floated around to define a stable api into the kernel for which stap is guaranteed not to break from one kernel version to the next. Mathieu proposed to keep a part of the stap self testing module into the kernel so that kernel developers can test themselves if they break something. Idea not practical, since it puts the burden on kernel community to assure that out of tree code keeps working. Problem is that stap doesn't use a predefined set of probe points. This is the whole idea behind the current stap design, that is, to get away from the static approach of dtrace, and to let you probe everything because of the use of the debuginfo. I.e. "API" is potentially the whole kernel code. However, the translator-generated and runtime-boilerplate code use a relatively small number of exported functions as the API. This is one reason why the runtime etc. is in fact remarkably stable from kernel version to kernel version. Maybe 1-3 changes per kernel release are needed. The tapsets are a different matter. Reorienting them with a preference for tracepoints will improve them too.


WRT keeping some stap tests in the kernel tree, one problem with this idea is that while the kernel developer can then instantly fix this hypothetical self-test module, and potentially make the self-test compile ("closing the feedback loop"), there is a hole in the feedback to the systemtap developers. Unless this "self-test module" is in fact complete, we still need to carry the fuller runtime/etc. code in systemtap, so we'd need to receive & redistribute those patches

- Mathieu mentioned gdb tracepoints: continuous debugging. Similar to systemtap idea. Collect info at breakpoints and continue the program. Code Sourcery working on that.



--
Elena Zannoni, Oracle
Senior Engineering Manager, Tools/Languages - Linux Engineering
Blog: http://blogs.oracle.com/ezannoni
Email: elena.zannoni@oracle.com


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]