This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
- From: William Cohen <wcohen at redhat dot com>
- To: Mark Wielaard <mjw at redhat dot com>
- Cc: "Frank Ch. Eigler" <fche at redhat dot com>, systemtap at sourceware dot org, Pratyush Anand <panand at redhat dot com>, Dave Long <dave dot long at linaro dot org>
- Date: Mon, 22 Jun 2015 09:53:28 -0400
- Subject: Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
- Authentication-results: sourceware.org; auth=none
- References: <55843208 dot 6030003 at redhat dot com> <y0mlhfflkl9 dot fsf at fche dot csb> <55848316 dot 4070109 at redhat dot com> <1434963644 dot 3016 dot 371 dot camel at bordewijk dot wildebeest dot org>
On 06/22/2015 05:00 AM, Mark Wielaard wrote:
> On Fri, 2015-06-19 at 17:01 -0400, William Cohen wrote:
>> However, I can't seem to get the stap module to be built with
>> debuginfo even with what seem like appropriate command line options.
>
> Are you sure? The following should generate debuginfo for the stap.ko
> you are creating:
>
> stap -k -p4 -vv -B CONFIG_DEBUG_INFO=y -e 'probe begin {log("hello"); exit(); }'
>
> If not, please post the -vv output so we can analyze what is going
> wrong.
>
>>> runtime/unwind.c:processCFI() works its way through the dwarf data
>>> with some protection against e.g. bad data, but it may be
>>> insufficient.
>>
>> I recall Mark making some changes to make elf handling more robust in
>> elfutils. I suspect similar robustification is missing somewhere in
>> the systemtap unwinder code.
>
> Maybe indeed. But I don't yet understand what is going in. Hopefully we
> can get a module with debuginfo to better analyse.
>
> Thanks,
>
> Mark
>
Hi Mark,
One of the things that I thought that might be tripping things up is that there might be previous builds that have been done without the -B CONFIG_DEBUG_INFO=y and the caching might be using. I tried building the module with:
/root/systemtap_write/install/bin/stap -B CONFIG_DEBUG_INFO=y -m l100 -p4 -vv -k systemtap.examples/memory/last_100_frees.stp -c "/root/systemtap_write/install/bin/stap -V" -d /root/systemtap_write/install/bin/stap --ldd >& l100.log
Below is "eu-readelf -S" output for the module which seems to have debuginfo in it:
$eu-readelf -S l100.ko
There are 44 section headers, starting at offset 0x301c60:
Section Headers:
[Nr] Name Type Addr Off Size ES Flags Lk Inf Al
[ 0] NULL 0000000000000000 00000000 00000000 0 0 0 0
[ 1] .text PROGBITS 0000000000000000 00000040 00011c7c 0 AX 0 0 4
[ 2] .rela.text RELA 0000000000000000 00302760 0000d1b8 24 42 1 8
[ 3] .fixup PROGBITS 0000000000000000 00011cbc 0000009c 0 AX 0 0 4
[ 4] .rela.fixup RELA 0000000000000000 0030f918 00000138 24 42 3 8
[ 5] .text.unlikely PROGBITS 0000000000000000 00011d58 00000088 0 AX 0 0 4
[ 6] .rela.text.unlikely RELA 0000000000000000 0030fa50 00000030 24 42 5 8
[ 7] .rodata PROGBITS 0000000000000000 00011de0 00000980 0 A 0 0 8
[ 8] .rela.rodata RELA 0000000000000000 0030fa80 000003a8 24 42 7 8
[ 9] __ex_table PROGBITS 0000000000000000 00012760 000000d0 0 A 0 0 8
[10] .rela__ex_table RELA 0000000000000000 0030fe28 00000270 24 42 9 8
[11] .modinfo PROGBITS 0000000000000000 00012830 000000b0 0 A 0 0 8
[12] __param PROGBITS 0000000000000000 000128e0 00000020 0 A 0 0 8
[13] .rela__param RELA 0000000000000000 00310098 00000048 24 42 12 8
[14] .rodata.str1.8 PROGBITS 0000000000000000 00012900 000bf218 1 AMS 0 0 8
[15] __mcount_loc PROGBITS 0000000000000000 000d1b18 00000570 0 A 0 0 8
[16] .rela__mcount_loc RELA 0000000000000000 003100e0 00001050 24 42 15 8
[17] .eh_frame PROGBITS 0000000000000000 000d2088 00002744 0 WA 0 0 8
[18] .rela.eh_frame RELA 0000000000000000 00311130 00001050 24 42 17 8
[19] .data PROGBITS 0000000000000000 000d47d0 001cc160 0 WA 0 0 8
[20] .rela.data RELA 0000000000000000 00312180 000ad460 24 42 19 8
[21] .data.unlikely PROGBITS 0000000000000000 002a0930 00000001 0 WA 0 0 1
[22] .stap_privilege PROGBITS 0000000000000000 002a0934 00000004 0 WA 0 0 4
[23] .gnu.linkonce.this_module PROGBITS 0000000000000000 002a0938 00000248 0 WA 0 0 8
[24] .rela.gnu.linkonce.this_module RELA 0000000000000000 003bf5e0 00000030 24 42 23 8
[25] .note.gnu.build-id NOTE 0000000000000000 002a0b80 00000024 0 A 0 0 4
[26] .bss NOBITS 0000000000000000 002a0ba8 00006798 0 WA 0 0 8
[27] .comment PROGBITS 0000000000000000 002a0ba8 00000087 1 MS 0 0 1
[28] .note.GNU-stack PROGBITS 0000000000000000 002a0c2f 00000000 0 0 0 1
[29] .debug_aranges PROGBITS 0000000000000000 002a0c2f 00000080 0 0 0 1
[30] .rela.debug_aranges RELA 0000000000000000 003bf610 00000078 24 42 29 8
[31] .debug_info PROGBITS 0000000000000000 002a0caf 0002ca16 0 0 0 1
[32] .rela.debug_info RELA 0000000000000000 003bf688 000404b8 24 42 31 8
[33] .debug_abbrev PROGBITS 0000000000000000 002cd6c5 00000b67 0 0 0 1
[34] .debug_line PROGBITS 0000000000000000 002ce22c 000063b3 0 0 0 1
[35] .rela.debug_line RELA 0000000000000000 003ffb40 00000030 24 42 34 8
[36] .debug_str PROGBITS 0000000000000000 002d45df 0000dea2 1 MS 0 0 1
[37] .debug_loc PROGBITS 0000000000000000 002e2481 00017ec5 0 0 0 1
[38] .rela.debug_loc RELA 0000000000000000 003ffb70 00030c60 24 42 37 8
[39] .debug_ranges PROGBITS 0000000000000000 002fa346 00007780 0 0 0 1
[40] .rela.debug_ranges RELA 0000000000000000 004307d0 00010440 24 42 39 8
[41] .shstrtab STRTAB 0000000000000000 00301ac6 00000195 0 0 0 1
[42] .symtab SYMTAB 0000000000000000 00440c10 00004188 24 43 552 8
[43] .strtab STRTAB 0000000000000000 00444d98 00003396 0 0 0 1
The original traceback was:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
Adjusting the addresses for the module by itself and Using addr2line the call trace maps back to:
a2e0
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:429
b3f4
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:1287
be80
/root/systemtap_write/install/share/systemtap/runtime/unwind.c:1518
c104
/root/systemtap_write/install/share/systemtap/runtime/stack.c:491
d7f4
/root/systemtap_write/install/share/systemtap/runtime/stack.c:591
de48
/tmp/stapUGCcwL/l100_src.c:867
I hope that provides a bit more insight into what is going wrong in the systemtap unwinder runtime.
-Will