Re: How to track the functions in self-written module using SystemTap?

[David Smith <dsmith at redhat dot com>]

On 12/06/2015 02:53 AM, Nan Xiao wrote:
> Hi David,
> 
>> (1)
>> I see I didn't explain well here. Once you copied your module to the
>> kernel's module directory, systemtap no longer would need a full path to
>> the module name - just "kex" should have worked.
> 
> Yeah, it worked!
> 
>> (2)
>> Hmm, it could mean that we just need to make sure that the debuginfo is
>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>> information for a module even if stap doesn't think it needs it.)
> 
> The output stills have error:
> 
> # stap -d /root/kernel/105.ops/kex.ko -v -e 'probe
> module("/root/kernel/105.ops/kex.ko").function("*") { printf("%s\n",
> print_backtrace()) }'
> Pass 1: parsed user script and 110 library script(s) using
> 220072virt/37580res/3088shr/34880data kb, in 150usr/10sys/162real ms.
> Pass 2: analyzed script: 6 probe(s), 1 function(s), 0 embed(s), 0
> global(s) using 221348virt/39612res/3832shr/36156data kb, in
> 20usr/60sys/114real ms.
> Pass 3: translated to C into
> "/tmp/stap2MezRE/stap_6395070fd6c3c139709af3ecba82092d_2576_src.c"
> using 221360virt/40108res/4196shr/36168data kb, in 10usr/50sys/66real
> ms.
> Pass 4: compiled C into
> "stap_6395070fd6c3c139709af3ecba82092d_2576.ko" in
> 2280usr/540sys/5099real ms.
> Pass 5: starting run.
> WARNING: no or bad debug frame hdr
> WARNING: No binary search table for debug frame, doing slow linear
> search for /root/kernel/105.ops/kex.ko
>  0xffffffffa0037000 : kex_init+0x0/0x0 [kex]
>  0xffffffff810020e8 (inexact)
>  0xffffffff810ed4ae (inexact)
>  0xffffffff81316880 (inexact)
>  0xffffffff810e9743 (inexact)
>  0xffffffff810ede66 (inexact)
>  0xffffffff81645909 (inexact)
> 
>  0xffffffffa03d70c4 : kex_cleanup+0x0/0x0 [kex]
>  0xffffffff810eb23b (inexact)
>  0xffffffff81641113 (inexact)
>  0xffffffff81014b12 (inexact)
>  0xffffffff81645909 (inexact)

Hmm. OK, let's make sure of a couple of things:

1) Make sure your kernel was compiled with frame pointer support (which
it should if it is a standard RHEL kernel):

# fgrep FRAME_POINTER /boot/config-`uname -r`
CONFIG_SCHED_OMIT_FRAME_POINTER=y
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y

2) Let's make sure your module still has debuginfo present. First, run
"file" on your module, making sure it says "not stripped". Then go a bit
deeper and run "readelf --sections" on your module - there should be
several sections that start with '.debug', including one called
'.debug_frame'.

> 
> Thanks!
> Best Regards
> Nan Xiao
> 
> 
> On Fri, Dec 4, 2015 at 8:15 PM, David Smith <dsmith@redhat.com> wrote:
>> On 12/04/2015 03:56 AM, Nan Xiao wrote:
>>> Hi Dave,
>>>
>>>> It looks like you've got a couple of options:
>>>
>>>> 1) Copy your module into the kernel module tree (located at
>>>> /lib/modules/`uname -r`/kernel) to use systemtap on it.
>>>
>>> I copy "kex.ko" into /lib/modules/`uname -r`/kernel, and it always
>>> executes error:
>>>
>>> # cd /lib/modules/`uname -r`/kernel
>>> # pwd
>>> /lib/modules/3.10.0-123.el7.x86_64.debug/kernel
>>> # ls
>>> arch  crypto  drivers  fs  kernel  kex.ko  lib  mm  net  sound
>>>
>>> # stap -v -e 'probe
>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>> { printf("%s\n", ppfunc()) }'
>>> Pass 1: parsed user script and 103 library script(s) using
>>> 214092virt/31440res/3036shr/29032data kb, in 160usr/60sys/226real ms.
>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>         source: probe
>>> module("/lib/modules/3.10.0-123.el7.x86_64.debug/kernel/kex.ko").function("*")
>>> { printf("%s\n", ppfunc()) }
>>>                       ^
>>>
>>> semantic error: no match
>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>> global(s) using 372028virt/33024res/3412shr/30124data kb, in
>>> 20usr/770sys/804real ms.
>>> Pass 2: analysis failed.  [man error::pass2]
>>>
>>> # stap -v -e 'probe module("kex.ko").function("*") { printf("%s\n",
>>> ppfunc()) }'                                    Pass 1: parsed user
>>> script and 103 library script(s) using
>>> 214096virt/31432res/3036shr/29036data kb, in 140usr/60sys/209real ms.
>>> semantic error: while resolving probe point: identifier 'module' at <input>:1:7
>>>         source: probe module("kex.ko").function("*") { printf("%s\n",
>>> ppfunc()) }
>>>                       ^
>>>
>>> semantic error: no match
>>> Pass 2: analyzed script: 0 probe(s), 0 function(s), 0 embed(s), 0
>>> global(s) using 372032virt/33016res/3412shr/30128data kb, in
>>> 40usr/880sys/929real ms.
>>> Pass 2: analysis failed.  [man error::pass2]
>>
>> I see I didn't explain well here. Once you copied your module to the
>> kernel's module directory, systemtap no longer would need a full path to
>> the module name - just "kex" should have worked.
>>
>>>> 2) Upgrade systemtap to at least 2.6. (I ran systemtap 2.6 on a RHEL 7.1
>>>> system, and the modules_out_of_tree.exp test case passed there.)
>>>
>>> I install the RHEL 7.2, and it really works! But it can't print actual
>>> stack traces:
>>>
>>> # stap -e 'probe module("/root/kernel/105.ops/kex.ko").function("*") {
>>> print_backtrace() }'
>>> WARNING: no or bad debug frame hdr
>>> WARNING: No binary search table for debug frame, doing slow linear
>>> search for /root/kernel/105.ops/kex.ko
>>>  0xffffffffa006b000 : kex_init+0x0/0x0 [kex]
>>>  0xffffffff810020e8 (inexact)
>>>  0xffffffff810ed4ae (inexact)
>>>  0xffffffff81316880 (inexact)
>>>  0xffffffff810e9743 (inexact)
>>>  0xffffffff810ede66 (inexact)
>>>  0xffffffff81645909 (inexact)
>>>  0xffffffffa02600c4 : kex_cleanup+0x0/0x0 [kex]
>>>  0xffffffff810eb23b (inexact)
>>>  0xffffffff81641113 (inexact)
>>>  0xffffffff81014b12 (inexact)
>>>  0xffffffff81645909 (inexact)
>>>
>>> So it means the module isn't built with debuginfo, right?
>>
>> Hmm, it could mean that we just need to make sure that the debuginfo is
>> loaded. Try adding the '-d  /root/kernel/105.ops/kex.ko' option to the
>> stap command line. (The '-d' option asks stap to load the symbol/unwind
>> information for a module even if stap doesn't think it needs it.)
>>
>> --
>> David Smith
>> dsmith@redhat.com
>> Red Hat
>> http://www.redhat.com
>> 256.217.0141 (direct)
>> 256.837.0057 (fax)


-- 
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)