This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
- From: William Cohen <wcohen at redhat dot com>
- To: systemtap at sourceware dot org, Mark Wielaard <mjw at redhat dot com>, Pratyush Anand <panand at redhat dot com>, Dave Long <dave dot long at linaro dot org>
- Date: Fri, 19 Jun 2015 12:28:55 -0400
- Subject: Re: last_100_frees.stp on aarch64 is crashing while doing sprint_ubacktrace()
- Authentication-results: sourceware.org; auth=none
- References: <55843208 dot 6030003 at redhat dot com>
On 06/19/2015 11:15 AM, William Cohen wrote:
> Pratyush has been working on uprobes support for aarch64 and I have been exercising it with systemtap. When running the tests on a locally built checkout of systemtap sometimes the last_100_frees.stp example causes the machine to crash. It looks like this is being triggered by the scrip doing the userspace backtrace. Any suggestions on how to further diagnose this? Other nuggets of information available in the output below?
>
> -Will
>
>
>
> Jun 19 10:36:45 apm-mustang-ev3-11 kernel: Unable to handle kernel paging request at virtual address fffffc0001707e90
> Jun 19 10:36:45 apm-mustang-ev3-11 kernel: pgd = fffffe01ddab0000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [fffffc0001707e90] *pgd=00000041ddd20003, *pud=00000041ddd20003, *pmd=00000041ddd20003, *pte=00000000000000\
> 00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Internal error: Oops: 96000007 [#1] SMP
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Modules linked in: stap_30b4cb5617d66b47c47d1ba687c18f92_2825(O) vfat fat xfs libcrc32c realtek [last unloa\
> ded: stap_30b4cb5617d66b47c47d1ba687c18f92_2814]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: CPU: 6 PID: 2826 Comm: stap Tainted: G W O 4.1.0-rc3+ #1
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Hardware name: AppliedMicro Mustang/Mustang, BIOS 1.1.0 Apr 22 2015
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: task: fffffe01dda97300 ti: fffffe00bd960000 task.ti: fffffe00bd960000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: PC is at processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: LR is at processCFI.constprop.119+0x75c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: pc : [<fffffdfffc4fa2e0>] lr : [<fffffdfffc4fa2c0>] pstate: 20000145
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: sp : fffffe00bd963b30
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x29: fffffe00bd963b30 x28: 0000000000005330
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x27: fffffdfffc501e88 x26: fffffdfffc76b119
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x25: 000003ffb3e36d1b x24: fffffc00016e04e0
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x23: fffffdfffc5034e8 x22: 0000000000000001
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x21: 000000000000001b x20: fffffdfffc501e8c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x19: fffffdfffc76b140 x18: 000003ffdafdd170
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x17: 0000000000675f50 x16: 000003ffb3e6d68c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x15: 0000000000000000 x14: fffffffffffff928
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x13: 0000000000000002 x12: fffffffffffff920
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x11: 0000000000000002 x10: fffffffffffff978
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x9 : 0000000000000040 x8 : 0000000000000039
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x7 : 000000000000004f x6 : 000000000000004f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x5 : 000000000000000e x4 : 000000000000004f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x3 : fffffdfffc76b11a x2 : 0000000000000228
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: x1 : fffffc0001707e90 x0 : 000000000000270f
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel:
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Process stap (pid: 2826, stack limit = 0xfffffe00bd960020)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Stack: (0xfffffe00bd963b30 to 0xfffffe00bd964000)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b20: bd963bd0 fffffe00 fc4fb3f8 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b40: fc502000 fffffdff 016e03b8 fffffc00 00000001 00000000 fc502510 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b60: 00000228 00000000 fc7537d0 fffffdff 016e04e0 fffffc00 b3e3b930 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3b80: fc761820 fffffdff 00005330 00000000 fc766969 fffffdff fc76b11a fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ba0: bd963c68 fffffe00 bd963c70 fffffe00 016e04e8 fffffc00 016e04f0 fffffc00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3bc0: bd963c98 fffffe00 bd963c64 fffffe00 bd963cb0 fffffe00 fc4fbe84 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3be0: fc753750 fffffdff 016e03b8 fffffc00 00000001 00000000 b3e36d1b 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c00: 016e03b8 fffffc00 00000000 00000400 00000004 00000000 bd960000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c20: 00ef0a00 fffffe00 ff070000 00000001 fc766954 fffffdff 00134c38 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c40: 00000004 00000000 00000004 00000008 fc766958 fffffdff b3e36d1b 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c60: 0000001b 00000001 fc766969 fffffdff fc766970 fffffdff fc76b0c1 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3c80: fc76b140 fffffdff b3e36600 000003ff 00005330 00000000 0000001e 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ca0: fc76248c fffffdff fc762488 fffffdff bd963d00 fffffe00 fc4fc108 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3cc0: 016e0000 fffffc00 00000004 00000000 bd963ed0 fffffe00 bd960000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ce0: 016e0000 fffffc00 fc7537d0 fffffdff b8ba6f28 fffffe00 fc753750 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d00: bd963d50 fffffe00 fc4fd7f8 fffffdff 016e0000 fffffc00 016e1ba8 fffffc00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d20: 00000005 00000000 fc790000 fffffdff 016e1000 fffffc00 00f42000 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d40: 016e1fb8 fffffc00 fc753750 fffffdff bd963dc0 fffffe00 fc4fde4c fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d60: fc790000 fffffdff fc5c4a68 fffffdff bd963ed0 fffffe00 fc790ab8 fffffdff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3d80: 73343777 00000006 d583b4a0 fffffe01 d583b420 fffffe01 00000001 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3da0: 00b5cf38 fffffe00 bd960000 fffffe00 016e1fb8 fffffc00 fff60a00 fffffe01
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3dc0: bd963e10 fffffe00 001a2af4 fffffe00 fc5c4a68 fffffdff b3e6d68c 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3de0: 00000000 00000000 bd963ed0 fffffe00 d583b400 fffffe01 001a2abc fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e00: bc2c0000 fffffe00 016e0000 fffffc00 bd963eb0 fffffe00 000972f0 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e20: 00000610 00000000 bd963ed0 fffffe00 ffffffff ffffffff b3e6d68c 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e40: 80000000 00000000 0000003c 00000000 f2000008 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e60: dafddb80 000003ff bd960000 fffffe00 00000000 00000000 00b5cff0 fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3e80: 00000000 00000000 00000001 00000000 00000000 00000003 00000003 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ea0: dafdda00 000003ff 000939d8 fffffe00 dafdd3a0 000003ff 0009388c fffffe00
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ec0: 00000600 00000000 00000030 00000000 0e8146e0 00000000 0040cdc0 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3ee0: 0e8148b4 00000000 0e8148b4 00000000 68676972 43282074 b3e74fbc 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f00: b3e74fac 000003ff ffffffff ffffffff 25252525 25252525 dafddcf8 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f20: ffffffff 00000000 0e814458 00000000 0000002d 00000000 dafddde0 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f40: cccb7300 d3140edb 005dff60 00000000 b3e6d68c 000003ff 00675f50 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f60: dafdd170 000003ff dafddb80 000003ff 00000030 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3f80: 0e8146e0 00000000 0e814850 00000000 00000064 00000000 0000012c 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fa0: 00000000 00000000 dafddb80 000003ff 00000000 00000000 dafdd3a0 000003ff
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fc0: b3e67680 000003ff dafdd3a0 000003ff b3e6d68c 000003ff 80000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: 3fe0: 00000003 00000000 ffffffff ffffffff 00000000 00000000 00000000 00000000
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Call trace:
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fa2e0>] processCFI.constprop.119+0x77c/0x8d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fb3f4>] unwind_frame.constprop.115+0x44c/0xe1c [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fbe80>] unwind+0xbc/0x148 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fc104>] _stp_stack_user_get+0x9c/0x1d8 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fd7f4>] probe_2718+0x234/0x524 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffdfffc4fde48>] stapiu_probe_prehandler+0x1d4/0x384 [stap_30b4cb5617d66b47c47d1ba687c18f92_2825]
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00001a2af0>] uprobe_notify_resume+0x3b4/0x8fc
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: [<fffffe00000972ec>] do_notify_resume+0x80/0x8c
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: Code: d2804501 9b017c42 91023001 8b011301 (a9401c26)
> Jun 19 10:36:46 apm-mustang-ev3-11 kernel: ---[ end trace 98f66aa70f9b298b ]---
>
Taking a look at the LR and PC values the snippet of the systemtap where things go wrong is in processCFI.constprop.119. The code has called and returned from get_uleb128 at a2bc. The processor continues on to a2e0. The ldp attempts to load the register pair from fffffc0001707e90 which causes the panic.
x24 is used as a base address in both a2dc and a2c0. The instruction at a2c0 only has an offset of 24 off of fffffc00016e04e0. x1 at a2e0 has offset of 279b0 (162,224 bytes) away. This is likely on another page.
0000000000009b64 <processCFI.constprop.119>: ...
a2b4: aa1303e1 mov x1, x19
a2b8: 9101a3a0 add x0, x29, #0x68
a2bc: 97ffd7df bl 238 <get_uleb128>
a2c0: 39406302 ldrb w2, [x24,#24]
a2c4: d284e1e1 mov x1, #0x270f // #9999
a2c8: f100801f cmp x0, #0x20
a2cc: 9a813000 csel x0, x0, x1, cc
a2d0: d2804501 mov x1, #0x228 // #552
a2d4: 9b017c42 mul x2, x2, x1
a2d8: 91023001 add x1, x0, #0x8c
a2dc: 8b011301 add x1, x24, x1, lsl #4
a2e0: a9401c26 ldp x6, x7, [x1]
a2e4: 8b001040 add x0, x2, x0, lsl #4
a2e8: 8b000300 add x0, x24, x0
a2ec: a9031c06 stp x6, x7, [x0,#48]
a2f0: f94037a6 ldr x6, [x29,#104]
a2f4: 52800021 mov w1, #0x1 // #1
-Will