Re: How to debug broken unwinding?

[Milian Wolff <mail at milianw dot de>]

On Dienstag, 13. Juni 2017 16:20:25 CEST Mark Wielaard wrote:
> Hi Milian,
> 
> On Thu, 2017-06-01 at 22:57 +0200, Milian Wolff wrote:
> > How should I debug this, or how can I report a good bug report for this? I
> > guess I could upload a perf archive and document the steps required to
> > build perf with libdw as the unwinder, as it allows to easily compare
> > libunwind and libdw for unwinding.
> 
> I am reading the rest of you messages in this thread now and it seems
> you did find the issue already. But the above would still be useful if
> you happen to have the instructions already written down. I was actually
> looking at perf recently and couldn't immediately figure out how to
> build it. It seems it depends on a whole kernel build. But if there is
> an easy way to just build perf (with a variant that uses libunwind and
> libdw) that would be useful in general I think.

Perf is, you could say, a two-fold software.

On one hand you have the kernel side of things, which does the PMU 
configuration and actual sampling and data recording.

One the other hand you have a bunch of userspace tooling that tells the kernel 
side to start recording, but - more importantly - to analyse the recorded 
data.

The first part in the kernel is pretty stable - no need to compile your own 
kernel. Esp. on a recent distro this is not required.

For the second side, compiling perf's userland is pretty easy to get the 
latest and greatest features. And it also allows us to compare libunwind vs. 
elfutils:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux \
    -b perf/core
cd linux/tools/perf
make # will output what dependencies where found
cp perf perf-libunwind
make NO_LIBUNWIND=1
cp perf perf-elfutils

# record data with either version
./perf record --call-graph dwarf <something>

Then you can use different "perf" versions to compare the unwinders. I usually 
diff the output of `perf script` or `perf report`, e.g.:

./perf-libunwind stat --inline > libunwind.txt
./perf-elfutils stat --inline > elfutils.txt
diff -u libunwind.txt elfutils.txt

Hope this helps!

> More in general debugging the unwinder is indeed too hard. There are
> some failures that are ignored because it isn't immediately clear
> whether they are fatal. e.g. you might be unable to recover some
> register value that then isn't ever used for a simple backtrace, or you
> might not find the CFI for a pc address, but want to use the fallback
> unwinder in that case. Maybe we should add a debug mode configuration
> that compiles in tracing output of the unwinder operations?

That would be helpful, I guess. Most notably for laymen like me who still have 
trouble understanding what exactly is going on here. If there would be some 
magic flag I could use (like libunwind's UNW_DEBUG_LEVEL=15) to generate a 
trace I could append to a bug report, that would be good to have!

Bye
-- 
Milian Wolff
mail@milianw.de
http://milianw.de