This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: dwarf unwinder (only works on i386/x86_64) - now with user space unwinding


Hi,

On Fri, 2009-04-17 at 16:05 +0200, Mark Wielaard wrote:
> I fixed up a couple of small things and enabled the dwarf unwinder for
> in-kernel unwinding (merge commit 7c2136cf), this also fixes bug #5748
> for which there is a testcase in testsuite/systemtap.context/context.exp
> (backtrace.tcl) that now has all calling functions in the trace (for a
> specific test module inserted). There can be some improvements to the
> code. The unwinder sometimes goes on after falling off the stack, when
> it should really use the fallback stack unwinder that was the default
> before. But in general the stack traces are more complete than before.
> There should be more tests written though.
> 
> Currently the dwarf unwinder is only enabled for i386 and x86_64
> [...]
> I am working on using the dwarf unwinder also for user space
> backtracing. First using the debug_frame tables that we also are using
> for the kernel case, but maybe switching to the eh_frame tables (it
> isn't clear which one is really the most accurate at the moment, we
> might need to consult both, but I am trying to avoid doing that for
> now).

I merged my user_unwind branch to master including some test cases.
uprobes_ustack.exp is the main one that does a couple of backtraces
through an exe and a shared library. This now works on i386 and x86_64.

This has only been lightly tested. But should just gently fail (you just
get no or a partial backtrace) in case of failures.

There is some cleanup to do with respect to the adjustStartAddress() in
unwind.c versus the _stp_module_relocate() and _stp_mod_sec_lookup() in
sym.c. See the comments/XXX/TODO there.

It should work against any probe that provides the CONTEXT with a user
pt_regs. But the test cases all use uprobe points for now. Any utrace
probe hook should also work, except that will almost always point into
the executables VDSO which we currently don't track and from which we
cannot unwind atm (bug #10080). timer probes should work since they
should provide a user pt_reg when interrupting a user process, which you
can test with context tapset function user_mode(). But they will only
work if you explicitly included the symbols and unwind data for the
libraries and executable (with -d) that gets interrupted.

The last point is a general gotcha that might surprise the user. We
might want to provide an easy option to include all shared libraries of
an executable a user wants to trace, since if you miss one with -d
backtraces are either unavailable or truncated.

There are a couple of new tapset functions included for inspecting user
backtraces and symbols. They are in ucontext-symbols.stp and
ucontext-unwind.stp. For now I have marked them EXPERIMENTAL (by adding
that string to their description) since they don't seem optimal yet, I
want to reorganize them (and their kernel context counterparts)
according to bug #6580. Suggestions welcome.

There are three somewhat obvious drawbacks with the new ucontext
functions at the moment. There really should be a way to associate a
ubacktrace() string with the task. Currently print_ustack() interprets
the given backtrace in the context of the current task, which can be
different between when ubacktrace() and when print_ustack() is called.

A special subcase of that is the end probe (where the process is most
likely already gone, and stapio is the current process). So something
like the following will not work properly:
stap -d /lib/libc.so.6 -ve \
'global b;
 probe process("/bin/ls").function("main").call { b = ubacktrace(); }
 probe end { print_ustack(b); }' -c '/bin/ls'
Which is a pity since you might want to do some filtering and statistics
in your end probe, but then you cannot get at the symbolic names of your
trace.

And finally ubacktrace() returns a string capped to MAXSTRINGLEN which
is only 128. On a 64bit architecture that means only 6 entries. So one
has to explicitly add -DMAXSTRINGLEN=something_much_bigger to handle
larger traces (print_ubacktrace() doesn't have that restriction, but
doesn't allow to capture the output and is more expensive since it does
a full symbol lookup immediately for the stack trace).

Ideas on how to make these things more intuitive for the user
appreciated.

Cheers,

Mark


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]