Re: Decoding stack in core file without correct libs?

[Jan Kratochvil <jan dot kratochvil at redhat dot com>]

On Thu, 13 Jul 2017 11:24:50 +0200, Pierre Ossman wrote:
> I'd like to see if there is a way to produce binaries so that gdb can walk
> the stack in a core dump even if the libraries gdb sees doesn't match the
> libraries when the core dump was generated.

Features described below are present at least in Fedora (and some of them on
RHEL-7).

Basic requirement is so-called build-id:
$ readelf -n /bin/bash
...
Displaying notes found at file offset 0x00000274 with length 0x00000024:
  Owner                 Data size	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: bad838505c566d846ec9571e4f592d108064cf92

Then Fedora GDB; not upstream GDB - this feature was never reviewed upstream
	[PATCH v12 00/32] Validate binary before use
	https://sourceware.org/ml/gdb-patches/2015-08/msg00590.html
will guide one to provide the matching packages for it:
$ gdb core.21935
...
Missing separate debuginfo for the main executable file
Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/379f708a44a91477b1407baf59c9bc1426d69e
Core was generated by `bc'.
...
# dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/07/379f708a44a91477b1407baf59c9bc1426d69e
...
  Installing  : bc-debuginfo-1.06.95-16.fc24.x86_64                                                                                  1/1 
...
$ gdb core.21935
...
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.24-9.fc25.x86_64 readline-6.3-8.fc24.x86_64
...

Installation of such libraries is sure a problem if you run on a system with
different libraries - you do not want to replace your system libraries just to
make a backtrace.  For such case there is ABRT - it will automatically build
a new chroot with libraries matching build-ids from the core file.
	https://developer.fedoraproject.org/tools/abrt/about.html

Otherwise one can also try store all the binaries into the core file:
	(ulimit -c unlimited;echo 0xff >/proc/self/coredump_filter;exec myapp appargs)
->
	eu-stack --core=core.21760

eu-stack from elfutils can extract debug information from binaries contained
in the core file (GDB cannot do that).  Although during my experiment it
succeeded with the libraries but it still failed to backtrace executable of
a different version on disk, there may be some bug.

Regular core file backtraced on a different system:
	$ eu-stack --core=core.21935
	PID 21935 - core
	TID 21935:
	#0  0x00007f88835d3a80
	eu-stack: dwfl_thread_getframes tid 21935 at 0x7f88835d3a80 in /usr/lib64/libc-2.24.so: Callback returned failure
coredump_filter-extended core file backtraced on a different system:
	eu-stack --core=core.21760 
	PID 21760 - core
	TID 21760:
	#0  0x00007ff4ee365a80 __read
	#1  0x00007ff4eeab4d1d rl_getc
	#2  0x00007ff4eeab559a rl_read_key
	#3  0x00007ff4eea9ee32 readline_internal_char
	#4  0x00007ff4eea9f595 readline
	#5  0x0000559a6c1ad47c
	eu-stack: dwfl_thread_getframes tid 21760 at 0x559a6c1ad47b in /usr/bin/bc: Callback returned failure
	^^^ some bug, #5 is in /usr/bin/bc which should be present in the core file.


Jan