Re: Unsuccessful attempts get aarch64 kernel backtraces support

[William Cohen <wcohen at redhat dot com>]

I noted in the -DDEBUG_UNWIND=1 output:

unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/pratyush/linux/vmlinux: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
unwind_frame:1194: /root/pratyush/linux/vmlinux: fde=0

It looks like the .debug_frame section is not available on aarch64.
To get a more complete picture of the differences between the
sections.  Extract out the section names on x86_64 and arm64 kernels
and then compare them:

'-' at beginning of line only on x86
'+' at beginning of line only on arm64

$ diff -u /tmp/xx /tmp/aa|grep [\-\+\]
-.apicdrivers
-.brk
-.data_nosave
-.debug_frame
+.eh_frame
+.head.text
-__init_rodata
+.init.eh_frame
+.init.init.text
+.init.rodata.str1.8
-.iommu_table
-.parainstructions
+.pecoff_edata_padding
-.smp_locks
-.tracedata
-.vvar
-.x86_cpu_dev.init
+.text.init


It looks like because the .debug_frame section is not available on the
arm64 kernel the systemtap runtime's dwarf unwinder tries to fall back
on using the eh_frame section.  The eh_frame is not sorted so it ends
up being a slow linear search, which the unwinder eventually gives up
on.

WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search for /root/pratyush/linux/vmlinux

Comparing the kernel sections to the section in userspace code see that the userspace has:

+.eh_frame_hdr

On arm64 the userspace looks like it is able to search because of the eh_frame_hdr.

# ~/systemtap_write/install/bin/stap -DDEBUG_UNWIND=1 -e 'probe process("./backtrace_test").function("c") {print_ubacktrace();}' -c ./backtrace_test
_stp_stack_unwind_one_user:462: STARTING user unwind
 0x4005c0 : c+0x0/0x8 [/root/backtrace_test]
_stp_stack_unwind_one_user:478: CONTINUING user unwind to depth 1
unwind:1478: pc=4005c0, 4005c0
unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/backtrace_test: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
_stp_search_unwind_hdr:775: binary search for 4005c0
_stp_search_unwind_hdr:840: fde off=4007a0
_stp_search_unwind_hdr:850: returning fde=fffffdfffc988b88 startLoc=4005c0
unwind_frame:1194: /root/backtrace_test: fde=fffffdfffc988b88
unwind_frame:1199: /root/backtrace_test: cie=fffffdfffc988af0
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
unwind_frame:1214: startLoc: 4005c0, endLoc: 4005c8
unwind_frame:1263: cie=fffffdfffc988af0 fde=fffffdfffc988b88 startLoc=4005c0 endLoc=4005c8, pc=4005c0
unwind_frame:1283: processCFI for CIE
processCFI:313: targetLoc=0 state->loc=4005c0
processCFI:519: map DW_CFA_def_cfa value 31 to reg_info idx 31
processCFI:521: DW_CFA_def_cfa reg=31
processCFI:530: DW_CFA_def_cfa_offset offs=0
processCFI:649: targetLoc=0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c0
processCFI:650: result: 1
unwind_frame:1291: processCFI for FDE
processCFI:313: targetLoc=4005c0 state->loc=4005c0
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=4005c0 state->loc=4005c0
processCFI:650: result: 1
unwind_frame:1310: cfa reg=31, off=0
unwind_frame:1315: cfa=3ffec904ca0 startLoc=3ffec904ca0, endLoc=3ffec904ca0
unwind_frame:1322: cie=fffffdfffc988af0 fde=fffffdfffc988b88
unwind_frame:1446: returning 0 (4005d4)
_stp_stack_unwind_one_user:503: ret=0 PC=4005d4 SP=3ffec904ca0
 0x4005d4 : b+0xc/0x14 [/root/backtrace_test]
_stp_stack_unwind_one_user:478: CONTINUING user unwind to depth 2
unwind:1478: pc=4005d3, 4005d4
unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/backtrace_test: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
_stp_search_unwind_hdr:775: binary search for 4005d3
_stp_search_unwind_hdr:840: fde off=4007b8
_stp_search_unwind_hdr:850: returning fde=fffffdfffc988ba0 startLoc=4005c8
unwind_frame:1194: /root/backtrace_test: fde=fffffdfffc988ba0
unwind_frame:1199: /root/backtrace_test: cie=fffffdfffc988af0
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
unwind_frame:1214: startLoc: 4005c8, endLoc: 4005dc
unwind_frame:1263: cie=fffffdfffc988af0 fde=fffffdfffc988ba0 startLoc=4005c8 endLoc=4005dc, pc=4005d3
unwind_frame:1283: processCFI for CIE
processCFI:313: targetLoc=0 state->loc=4005c8
processCFI:519: map DW_CFA_def_cfa value 31 to reg_info idx 31
processCFI:521: DW_CFA_def_cfa reg=31
processCFI:530: DW_CFA_def_cfa_offset offs=0
processCFI:649: targetLoc=0 state->loc=4005c8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005c8
processCFI:650: result: 1
unwind_frame:1291: processCFI for FDE
processCFI:313: targetLoc=4005d3 state->loc=4005c8
advance_loc:241: state->loc=4005cc
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005d3 state->loc=4005cc
processCFI:650: result: 1
processCFI:530: DW_CFA_def_cfa_offset offs=10
processCFI:649: targetLoc=4005d3 state->loc=4005cc
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 29 to reg_info idx 29
set_offset_rule:259: reg=1d, where=2, svalue=fffffffffffffff0
processCFI:649: targetLoc=4005d3 state->loc=4005cc
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 30 to reg_info idx 30
set_offset_rule:259: reg=1e, where=2, svalue=fffffffffffffff8
processCFI:649: targetLoc=4005d3 state->loc=4005cc
processCFI:650: result: 1
advance_loc:241: state->loc=4005d0
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005d3 state->loc=4005d0
processCFI:650: result: 1
processCFI:566: map DW_CFA_def_cfa_register value 29 to reg_info idx 29 (29)
processCFI:649: targetLoc=4005d3 state->loc=4005d0
processCFI:650: result: 1
advance_loc:241: state->loc=4005d8
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005d3 state->loc=4005d8
processCFI:650: result: 1
unwind_frame:1310: cfa reg=29, off=10
unwind_frame:1315: cfa=3ffec904cb0 startLoc=3ffec904ca0, endLoc=3ffec904cb0
unwind_frame:1322: cie=fffffdfffc988af0 fde=fffffdfffc988ba0
unwind_frame:1436: set register 29 to 3ffec904cb0
unwind_frame:1436: set register 30 to 4005e8
unwind_frame:1446: returning 0 (4005e8)
_stp_stack_unwind_one_user:503: ret=0 PC=4005e8 SP=3ffec904cb0
 0x4005e8 : a+0xc/0x14 [/root/backtrace_test]
_stp_stack_unwind_one_user:478: CONTINUING user unwind to depth 3
unwind:1478: pc=4005e7, 4005e8
unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/backtrace_test: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
_stp_search_unwind_hdr:775: binary search for 4005e7
_stp_search_unwind_hdr:840: fde off=4007e0
_stp_search_unwind_hdr:850: returning fde=fffffdfffc988bc8 startLoc=4005dc
unwind_frame:1194: /root/backtrace_test: fde=fffffdfffc988bc8
unwind_frame:1199: /root/backtrace_test: cie=fffffdfffc988af0
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
unwind_frame:1214: startLoc: 4005dc, endLoc: 4005f0
unwind_frame:1263: cie=fffffdfffc988af0 fde=fffffdfffc988bc8 startLoc=4005dc endLoc=4005f0, pc=4005e7
unwind_frame:1283: processCFI for CIE
processCFI:313: targetLoc=0 state->loc=4005dc
processCFI:519: map DW_CFA_def_cfa value 31 to reg_info idx 31
processCFI:521: DW_CFA_def_cfa reg=31
processCFI:530: DW_CFA_def_cfa_offset offs=0
processCFI:649: targetLoc=0 state->loc=4005dc
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005dc
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005dc
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005dc
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005dc
processCFI:650: result: 1
unwind_frame:1291: processCFI for FDE
processCFI:313: targetLoc=4005e7 state->loc=4005dc
advance_loc:241: state->loc=4005e0
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005e7 state->loc=4005e0
processCFI:650: result: 1
processCFI:530: DW_CFA_def_cfa_offset offs=10
processCFI:649: targetLoc=4005e7 state->loc=4005e0
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 29 to reg_info idx 29
set_offset_rule:259: reg=1d, where=2, svalue=fffffffffffffff0
processCFI:649: targetLoc=4005e7 state->loc=4005e0
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 30 to reg_info idx 30
set_offset_rule:259: reg=1e, where=2, svalue=fffffffffffffff8
processCFI:649: targetLoc=4005e7 state->loc=4005e0
processCFI:650: result: 1
advance_loc:241: state->loc=4005e4
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005e7 state->loc=4005e4
processCFI:650: result: 1
processCFI:566: map DW_CFA_def_cfa_register value 29 to reg_info idx 29 (29)
processCFI:649: targetLoc=4005e7 state->loc=4005e4
processCFI:650: result: 1
advance_loc:241: state->loc=4005ec
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=4005e7 state->loc=4005ec
processCFI:650: result: 1
unwind_frame:1310: cfa reg=29, off=10
unwind_frame:1315: cfa=3ffec904cc0 startLoc=3ffec904cb0, endLoc=3ffec904cc0
unwind_frame:1322: cie=fffffdfffc988af0 fde=fffffdfffc988bc8
unwind_frame:1436: set register 29 to 3ffec904cc0
unwind_frame:1436: set register 30 to 400604
unwind_frame:1446: returning 0 (400604)
_stp_stack_unwind_one_user:503: ret=0 PC=400604 SP=3ffec904cc0
 0x400604 : main+0x14/0x18 [/root/backtrace_test]
_stp_stack_unwind_one_user:478: CONTINUING user unwind to depth 4
unwind:1478: pc=400603, 400604
unwind:1521: trying debug_frame
unwind_frame:1178: Module /root/backtrace_test: no unwind frame data
unwind:1525: debug_frame failed: -5, trying eh_frame
_stp_search_unwind_hdr:775: binary search for 400603
_stp_search_unwind_hdr:840: fde off=400808
_stp_search_unwind_hdr:850: returning fde=fffffdfffc988bf0 startLoc=4005f0
unwind_frame:1194: /root/backtrace_test: fde=fffffdfffc988bf0
unwind_frame:1199: /root/backtrace_test: cie=fffffdfffc988af0
parse_fde_cie:156: map retAddrReg value 30 to reg_info idx 30
unwind_frame:1214: startLoc: 4005f0, endLoc: 400608
unwind_frame:1263: cie=fffffdfffc988af0 fde=fffffdfffc988bf0 startLoc=4005f0 endLoc=400608, pc=400603
unwind_frame:1283: processCFI for CIE
processCFI:313: targetLoc=0 state->loc=4005f0
processCFI:519: map DW_CFA_def_cfa value 31 to reg_info idx 31
processCFI:521: DW_CFA_def_cfa reg=31
processCFI:530: DW_CFA_def_cfa_offset offs=0
processCFI:649: targetLoc=0 state->loc=4005f0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005f0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005f0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005f0
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=0 state->loc=4005f0
processCFI:650: result: 1
unwind_frame:1291: processCFI for FDE
processCFI:313: targetLoc=400603 state->loc=4005f0
advance_loc:241: state->loc=4005f4
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=400603 state->loc=4005f4
processCFI:650: result: 1
processCFI:530: DW_CFA_def_cfa_offset offs=20
processCFI:649: targetLoc=400603 state->loc=4005f4
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 29 to reg_info idx 29
set_offset_rule:259: reg=1d, where=2, svalue=ffffffffffffffe0
processCFI:649: targetLoc=400603 state->loc=4005f4
processCFI:650: result: 1
processCFI:627: map DW_CFA_offset value 30 to reg_info idx 30
set_offset_rule:259: reg=1e, where=2, svalue=ffffffffffffffe8
processCFI:649: targetLoc=400603 state->loc=4005f4
processCFI:650: result: 1
advance_loc:241: state->loc=4005f8
processCFI:617: DW_CFA_advance_loc
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:566: map DW_CFA_def_cfa_register value 29 to reg_info idx 29 (29)
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
processCFI:322: DW_CFA_nop
processCFI:649: targetLoc=400603 state->loc=4005f8
processCFI:650: result: 1
unwind_frame:1310: cfa reg=29, off=20
unwind_frame:1315: cfa=3ffec904ce0 startLoc=3ffec904cc0, endLoc=3ffec904ce0
unwind_frame:1322: cie=fffffdfffc988af0 fde=fffffdfffc988bf0
unwind_frame:1436: set register 29 to 3ffec904ce0
unwind_frame:1436: set register 30 to 3ff8367f68c
unwind_frame:1446: returning 0 (3ff8367f68c)
_stp_stack_unwind_one_user:503: ret=0 PC=3ff8367f68c SP=3ffec904ce0
 0x3ff8367f68c [/usr/lib64/libc-2.22.so+0x1f68c/0x190000]
_stp_stack_unwind_one_user:478: CONTINUING user unwind to depth 5
unwind:1478: pc=3ff8367f68b, 3ff8367f68c
unwind:1517: No module found for pc=3ff8367f68b
_stp_stack_unwind_one_user:503: ret=-22 PC=3ff8367f68c SP=3ffec904ce0
WARNING: Missing unwind data for a module, rerun with 'stap -d /usr/lib64/libc-2.22.so'


-Will