This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: Problem with vfs probe on Proxmox kernel
On 01/26/2017 12:42 PM, Adam Guderski wrote:
> On 26/01/17 18:37, David Smith wrote:
>> On 01/25/2017 04:44 PM, Adam Guderski wrote:
>>> Hello,
>>>
>>> I've got problem running sample traceio.stp script (obtained from
>>> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/SystemTap_Beginners_Guide/#traceio2)
>>> on Proxmox VE 4.4 system. Proxmox is a Debian-based Linux distro which
>>> uses a custom Ubuntu-based kernel. The Proxmox repository does not
>>> provide a kernel with debug symbols, so I had to compile it from
>>> provided sources - version 4.4.35-2-pve, to be specific. Debian repos
>>> (jessie) provide systemtap in version 2.9, which from what I read does
>>> not support 4.4 kernels, so I compiled systemtap from sources (from
>>> master branch, version 3.1/0.159).
>>>
>>> With such setup, I'm able to run sample "hello world" stap script and
>>> couple of others, but all scripts with vfs probes fail. I attach full
>>> verbose (-vvv) output here: http://pastebin.com/DiWMsrtr, but I think
>>> the main error can be explained below. What can I do about it? I
>>> searched for couple of hours but did not find anything that would help.
>>
>> As the following states:
>>
>>> # stap -v traceio2.stp 0xfb00
>>> Pass 1: parsed user script and 119 library scripts using
>>> 71608virt/43716res/4464shr/39904data kb, in 140usr/0sys/132real ms.
>>> semantic error: while resolving probe point: identifier 'kernel' at
>>> /usr/local/share/systemtap/tapset/linux/vfs.stp:987:19
>>> source: probe vfs.write = kernel.function("vfs_write")
>>> ^
>>>
>>> semantic error: missing x86_64 kernel/module debuginfo [man
>>> warning::debuginfo] under '/lib/modules/4.4.35-2-pve/build'
>>
>> Note that a systemtap "hello world" program doesn't need kernel
>> debuginfo, but a script using vfs probes does need kernel debuginfo.
>> Does any script that needs debuginfo work, or is it just vfs probes? To
>> test, try 'stap -l 'kernel.function("*_write")'. You should see items
>> like 'sys_write' and 'vfs_write' in the output.
>
> Sadly, the command "stap -l 'kernel.function("*_write")'" returns
> nothing at all.
That's actually good. This definitely means that stap can't find any
debuginfo (vs. the problem being stap found the debuginfo but didn't
find vfs_write() in the debuginfo).
>> systemtap is expecting to find the kernel's debuginfo at
>> /lib/modules/4.4.35-2-pve/build. Is it there? If not, you'll either need
>> to move it there or use the 'SYSTEMTAP_DEBUGINFO_PATH' environment
>> variable to tell systemtap where you've put the debuginfo.
>
> Could you elaborate how debuginfo files look like? This is what I've got
> in the directory:
For unstripped userspace exes, the debuginfo is present in the exe. If
you probe an unstripped userspace exe, systemtap shouldn't have any
trouble finding the debuginfo. To make the exes smaller, the debuginfo
can be extracted and put in another file, typically in
/usr/lib/debug/EXEPATH.debug. So, if you install the debuginfo for
/bin/ls, you'll find its debuginfo in /usr/lib/debug/bin/ls.debug.
For kernels, things are a bit more tricky. You typically boot off a
compressed stripped kernel image, most likely called
/boot/vmlinuz-KERNEL_VER (where 'KERNEL_VER' is the kernel version). The
debuginfo is typically stored in an unstripped uncompressed kernel file
down in /usr/lib/debug. Here's on a Fedora 25 system I see:
====
# ls -l /boot/vmlinuz-4.9.5-200.fc25.x86_64
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
-rwxr-xr-x. 1 root root 6891144 Jan 20 07:43
/boot/vmlinuz-4.9.5-200.fc25.x86_64
-rwxr-xr-x. 1 root root 541141280 Jan 20 07:48
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
# file /boot/vmlinuz-4.9.5-200.fc25.x86_64
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux
/boot/vmlinuz-4.9.5-200.fc25.x86_64: Linux kernel
x86 boot executable bzImage, version 4.9.5-200.fc25.x86_64
(mockbuild@bkernel02.phx2.fedoraproject.o, RO-rootFS, swap_dev 0x6,
Normal VGA
/usr/lib/debug/lib/modules/4.9.5-200.fc25.x86_64/vmlinux: ELF 64-bit LSB
executable, x86-64, version 1 (SYSV), statically linked,
BuildID[sha1]=05af643a7230786a9f0fe4fc1c05db0be64ef44c, not stripped
====
As you can see the vmlinux file is much bigger than the vmlinuz file,
since it is unstripped and uncompressed.
Now those paths are all for distro files, and you've got a self-built
kernel. But, the principle is still the same.
> # ls -la /lib/modules/4.4.35-2-pve/build/
> total 1564
> drwxr-xr-x 26 root root 4096 Jan 22 02:09 .
> drwxr-xr-x 3 root root 4096 Jan 18 00:52 ..
> drwxr-xr-x 33 root root 4096 Jan 22 02:09 arch
> drwxr-xr-x 3 root root 4096 Jan 22 02:09 block
> drwxr-xr-x 2 root root 4096 Jan 22 02:09 certs
> -rw-r--r-- 1 root root 189740 Jan 22 01:29 .config
> drwxr-xr-x 4 root root 4096 Jan 22 02:09 crypto
> drwxr-xr-x 127 root root 4096 Jan 22 02:09 drivers
> drwxr-xr-x 2 root root 4096 Jan 22 02:09 firmware
> drwxr-xr-x 74 root root 4096 Jan 22 02:09 fs
> drwxr-xr-x 30 root root 4096 Jan 22 02:09 include
> drwxr-xr-x 2 root root 4096 Jan 22 02:09 init
> drwxr-xr-x 2 root root 4096 Jan 22 02:09 ipc
> -rw-r--r-- 1 root root 2622 Dec 22 08:43 Kbuild
> -rw-r--r-- 1 root root 252 Dec 22 08:43 Kconfig
> drwxr-xr-x 15 root root 4096 Jan 22 02:09 kernel
> drwxr-xr-x 12 root root 4096 Jan 22 02:09 lib
> -rw-r--r-- 1 root root 56164 Jan 22 00:46 Makefile
> drwxr-xr-x 3 root root 4096 Jan 22 02:09 mm
> -rw-r--r-- 1 root root 1218952 Jan 22 01:29 Module.symvers
> drwxr-xr-x 60 root root 4096 Jan 22 02:09 net
> drwxr-xr-x 16 root root 4096 Jan 22 02:09 samples
> drwxr-xr-x 13 root root 20480 Jan 22 02:09 scripts
> drwxr-xr-x 9 root root 4096 Jan 22 02:09 security
> drwxr-xr-x 23 root root 4096 Jan 22 02:09 sound
> drwxr-xr-x 10 root root 4096 Jan 22 02:09 spl
> drwxr-xr-x 21 root root 4096 Jan 22 02:09 tools
> drwxr-xr-x 8 root root 4096 Jan 22 02:09 ubuntu
> drwxr-xr-x 2 root root 4096 Jan 22 02:09 usr
> drwxr-xr-x 4 root root 4096 Jan 22 02:09 virt
> drwxr-xr-x 13 root root 4096 Jan 22 02:09 zfs
>
> Am I missing something? Are debuginfo files output somewhere else during
> compilation from source?
Based on the paths stored when compiling your kernel, systemtap is
looking for the vmlinux file in '/lib/modules/4.4.35-2-pve/build/'.
Obviously it isn't there. Can you search around and see if you can find
it and link it to its proper place?
--
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)