Re: How to track the functions in self-written module using SystemTap?

[Nan Xiao <xiaonan830818 at gmail dot com>]

Hi David,

> 1) Make sure your kernel was compiled with frame pointer support (which
> it should if it is a standard RHEL kernel):

> # fgrep FRAME_POINTER /boot/config-`uname -r`
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y

Yes, all the CONFIG options are "yes".

> 2) Let's make sure your module still has debuginfo present. First, run
> "file" on your module, making sure it says "not stripped".

Yes, it is "not stripped".

> Then go a bit deeper and run "readelf --sections" on your module - there should be
> several sections that start with '.debug', including one called
> '.debug_frame'.

Below are all the sections with `.debug`:

......
[19] .debug_info       PROGBITS         0000000000000000  000008e0
       000000000000b6d9  0000000000000000           0     0     1
  [20] .rela.debug_info  RELA             0000000000000000  00014fe0
       000000000000ed00  0000000000000018          36    19     8
  [21] .debug_abbrev     PROGBITS         0000000000000000  0000bfb9
       00000000000007cc  0000000000000000           0     0     1
  [22] .debug_loc        PROGBITS         0000000000000000  0000c785
       0000000000000324  0000000000000000           0     0     1
  [23] .rela.debug_loc   RELA             0000000000000000  00023ce0
       0000000000000570  0000000000000018          36    22     8
  [24] .debug_aranges    PROGBITS         0000000000000000  0000caa9
       0000000000000070  0000000000000000           0     0     1
  [25] .rela.debug_arang RELA             0000000000000000  00024250
       0000000000000078  0000000000000018          36    24     8
  [26] .debug_ranges     PROGBITS         0000000000000000  0000cb19
       0000000000000040  0000000000000000           0     0     1
  [27] .rela.debug_range RELA             0000000000000000  000242c8
       0000000000000090  0000000000000018          36    26     8
  [28] .debug_line       PROGBITS         0000000000000000  0000cb59
       0000000000000a8e  0000000000000000           0     0     1
  [29] .rela.debug_line  RELA             0000000000000000  00024358
       0000000000000048  0000000000000018          36    28     8
  [30] .debug_str        PROGBITS         0000000000000000  0000d5e7
       00000000000069fc  0000000000000001  MS       0     0     1
  [31] .comment          PROGBITS         0000000000000000  00013fe3
       000000000000005a  0000000000000001  MS       0     0     1
  [32] .note.GNU-stack   PROGBITS         0000000000000000  0001403d
       0000000000000000  0000000000000000           0     0     1
  [33] .debug_frame      PROGBITS         0000000000000000  00014040
       00000000000000f0  0000000000000000           0     0     8
  [34] .rela.debug_frame RELA             0000000000000000  000243a0
       00000000000000f0  0000000000000018          36    33     8
......

Thanks!
Best Regards
Nan Xiao


On Tue, Dec 8, 2015 at 3:53 AM, David Smith <dsmith@redhat.com> wrote:
> On 12/06/2015 02:53 AM, Nan Xiao wrote:
>> Hi David,
>>
>>> (1)
>>> I see I didn't explain well here. Once you copied your module to the
>>> kernel's module directory, systemtap no longer would need a full path to
>>> the module name - just "kex" should have worked.
>>
>> Yeah, it worked!
>>
>>> (2)
>>> Hmm, it could mean that we just need to make sure that the debuginfo is
>>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>>> information for a module even if stap doesn't think it needs it.)
>>
>> The output stills have error:
>>
>> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
>> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
>> print_backtrace()) }'
>> Pass 1: parsed user script and 110 library script(s) using
>> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms.
>> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
>> global(s) using 221348virt/39612res/3832shr/36156data kb, in
>> 20usr/60sys/114real ms.
>> Pass 3: translated to C into
>> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c"
>> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real
>> ms.
>> Pass 4: compiled C into
>> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in
>> 2280usr/540sys/5099real ms.
>> Pass 5: starting run.
>> WARNING: no or bad debug frame hdr
>> WARNING: No binary search table for debug frame, doing slow linear
>> search for /root/kernel/105.ops/kex.ko
>>  0xffffffffa0037000 : kex_init+0x0/0x0 [kex]
>>  0xffffffff810020e8 (inexact)
>>  0xffffffff810ed4ae (inexact)
>>  0xffffffff81316880 (inexact)
>>  0xffffffff810e9743 (inexact)
>>  0xffffffff810ede66 (inexact)
>>  0xffffffff81645909 (inexact)
>>
>>  0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex]
>>  0xffffffff810eb23b (inexact)
>>  0xffffffff81641113 (inexact)
>>  0xffffffff81014b12 (inexact)
>>  0xffffffff81645909 (inexact)
>
> Hmm. OK, let's make sure of a couple of things:
>
> 1) Make sure your kernel was compiled with frame pointer support (which
> it should if it is a standard RHEL kernel):
>
> # fgrep FRAME_POINTER /boot/config-`uname -r`
> CONFIG_SCHED_OMIT_FRAME_POINTER=y
> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
>
> 2) Let's make sure your module still has debuginfo present. First, run
> "file" on your module, making sure it says "not stripped". Then go a bit
> deeper and run "readelf --sections" on your module - there should be
> several sections that start with '.debug', including one called
> '.debug_frame'.
>
>>
>> Thanks!
>> Best Regards
>> Nan Xiao
>>
>>
>> On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote:
>>> On 12/04/2015 03:56 AM, Nan Xiao wrote:
>>>> Hi Dave,
>>>>
>>>>> It looks like you've got a couple of options:
>>>>
>>>>> 1) Copy your module into the kernel module tree (located at
>>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>>>>
>>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
>>>> executes error:
>>>>
>>>> # cd /lib/modules/`uname -r`/kernel
>>>> # pwd
>>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
>>>> # ls
>>>> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
>>>>
>>>> # stap -v -e 'probe
>>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>>> { printf("%s\n", ppfunc()) }'
>>>> Pass 1: parsed user script and 103 library script(s) using
>>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
>>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>>         source: probe
>>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>>> { printf("%s\n", ppfunc()) }
>>>>                       ^
>>>>
>>>> semantic error: no match
>>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in
>>>> 20usr/770sys/804real ms.
>>>> Pass 2: analysis failed.  [man error::pass2]
>>>>
>>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
>>>> ppfunc()) }'                                    Pass 1: parsed user
>>>> script and 103 library script(s) using
>>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
>>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>>         source: probe module("kex.ko").function("*") { printf("%s\n",
>>>> ppfunc()) }
>>>>                       ^
>>>>
>>>> semantic error: no match
>>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in
>>>> 40usr/880sys/929real ms.
>>>> Pass 2: analysis failed.  [man error::pass2]
>>>
>>> I see I didn't explain well here. Once you copied your module to the
>>> kernel's module directory, systemtap no longer would need a full path to
>>> the module name - just "kex" should have worked.
>>>
>>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>>>>> system, and the modules_out_of_tree.exp test case passed there.)
>>>>
>>>> I install the RHEL 7.2, and it really works! But it can't print actual
>>>> stack traces:
>>>>
>>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
>>>> print_backtrace() }'
>>>> WARNING: no or bad debug frame hdr
>>>> WARNING: No binary search table for debug frame, doing slow linear
>>>> search for /root/kernel/105.ops/kex.ko
>>>>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>>>>  0xffffffff810020e8 (inexact)
>>>>  0xffffffff810ed4ae (inexact)
>>>>  0xffffffff81316880 (inexact)
>>>>  0xffffffff810e9743 (inexact)
>>>>  0xffffffff810ede66 (inexact)
>>>>  0xffffffff81645909 (inexact)
>>>>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>>>>  0xffffffff810eb23b (inexact)
>>>>  0xffffffff81641113 (inexact)
>>>>  0xffffffff81014b12 (inexact)
>>>>  0xffffffff81645909 (inexact)
>>>>
>>>> So it means the module isn't built with debuginfo, right?
>>>
>>> Hmm, it could mean that we just need to make sure that the debuginfo is
>>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>>> information for a module even if stap doesn't think it needs it.)
>>>
>>> --
>>> David Smith
>>> dsmith@redhat.com
>>> Red Hat
>>> http://www.redhat.com
>>> 256.217.0141 (direct)
>>> 256.837.0057 (fax)
>
>
> --
> David Smith
> dsmith@redhat.com
> Red Hat
> http://www.redhat.com
> 256.217.0141 (direct)
> 256.837.0057 (fax)