Re: kernel summit session on systemtap

[Theodore Tso <tytso at mit dot edu>]

On Wed, Sep 17, 2008 at 08:49:02PM -0400, Frank Ch. Eigler wrote:
> Right.  We (systemtap and associated tools folks) are working on
> - improving quality (benefits) of dwarf
> - shrinking dwarf dramatically
> - if all else fails, dwarf subsetting

What do you think is the timeline for this happening?  I assume this
requires changes to gcc, right?  So would an estimate of 6-9 months,
minimum, be a fair one --- and that's assuming that a new gcc with
more aggressive optimizations that may or may not kernel the kernel
will be immediately usable for kernel builds (regardless of whether
the fault lies with the ANSI standards committeee, gcc developers'
boneheadedness, or kernel developers' boneheadedness, it's not always
the case that a new gcc is immediately trusted or even suitable for
use with the kernel.)

Given that, it might be a good idea to pursue a dwarf subsetting idea;
otherwise, many other tracing tools may make great strides and
improvements, while this particular shortcoming of systemtap means
that many kernel developers will find other ways of getting the
functionality they need and stop paying attention to Systemtap in
favor of tools that require debuginfo information.

As you may have seen by some of the horrible hacks that Steven Rostedt
pursued in order to work how gcc inserts mcount code for profiling, or
some of the other "interesting" uses/abuses of compiler toolchain,
there is no shortage of gross build kludges in the kernel.  So
something which some how manages to trick the current compiler into
emitting DWARF information so that function parameters can be decoded
could provide substantial benefits over simply using a limited set of
markers.

> Changing kernel build flags is of course possible, but it is not our
> place to mandate that.

Well, both of these ideas --- some kind of build-time kludge to create
limited DWARF information quickly using current compiler technology
and a different set of compiler optimization flags to make Systemtap
more useful --- are patches that would need to be applied to the
kernel, yes.  So it's not a question of mandating them, but rather
suggesting them as options that might make the use of Systemtap more
palatable in the short term.

In the past I've submitted patches gave an option to the kernel's
"make install" to strip out the debuginfo so that the partition
containing /lib wouldn't run out of space.  Right now I manually
install the full set of module files in /usr/lib/debug/lib/modules/... via:

	make INSTALL_MOD_STRIP=1 install_modules
	make INSTALL_MOD_PATH=/usr/lib/debug install_modules

Yeah, I should strip out the code/data segments out of what's in
/usr/lib/debug, but as a percentage of the bloat-o-rama which is the
debuginfo information, the savings just hasn't been big enough for me
to find the motivation to hack in the kernel build extension to do
this for me.

So submitting patches to make systemtap more useful isn't something
you can "mandate", but there are some of us who have used Systemtap
enough who would be willing to champion such patches, if we got some
help in crafting them in the first place.

I can implement the patch to reduce kernel optimization levels to make
debuginfo more helpful; but tricking the compiler to quickly generate
a limited set of DWARF informatoin is something I would need help
doing.

> > [...] Quite frankly, these days the main reason why I haven't been
> > 	playing with Systemtap much lately is because I'm tired of
> > 	waiting for compiles to complete when compiling with
> > 	debuginfo.  [...]
> 
> (By the way, do you build distro-style kernels on your laptop, with
> allmodconfig or somesuch, or something more linus-sized?)

I do both.  The distro-style kernels are the ones that I build with
debuginfo information, and it's been useful for playing around with
systemtap, but the moment I need do any serious development work, I
tend to fall back to a limited subset of compile options, generally
without any modules, and printk debugging.  Once we get a useful
circular buffer, I'd probably start logging to the circular buffer and
use grep as the alternative to systemtap or printk debugging.  As a
result, I've had no motivation to create any tapsets, since at least
for my own personal needs, the costs of creating the debuginfo so that
SystemTap would be useful for my personal needs just far outweighs the
benefits.

> Your information is slightly obsolete.  We just added some such
> automation, and can do more.

Glad to hear it.  I suspect then that this page:

     http://sources.redhat.com/systemtap/wiki/SystemtapDtraceComparison

is also slightly out of date.

							- Ted