Re: SystemTap 3.0 doesn't work on Ubuntu 14.04.4 LTS

[Josh Stone <jistone at redhat dot com>]

On 03/29/2016 02:53 PM, Aleksander Alekseev wrote:
> Hello
> 
> Thanks for your replies!
> 
>> What issues do you have?
>> It is certainly supposed to build against elfutils 0.165.
> 
> `make` fails like this:
> 
> http://pastebin.com/raw/afHGbFpE
> 
>> As noted in that answer, can you provide "objdump -d" around those
>> addresses?
> 
> Could you please explain how to find reported address (e.g.
> 0xffffffff81205870) in objdump output? Here is a full output of `sudo
> objdump -dr /boot/vmlinuz-4.2.0-27-generic` command:
> 
> https://www.dropbox.com/s/w9iywua0c6pdkju/vmlinux.decoded.gz?dl=0
> 
> I hope at list this is a right assembly code? :)

Hmm, right, I haven't actually tried that in a while.  objdump doesn't
make it easy to output a specific function, but there's --start-address
and --stop-address if you can figure out the right offsets.

I think objdump does try to decompress vmlinuz, but the results have a
lot of "(bad)" lines, so I don't really trust this.  You should use the
kernel's script to extract it first:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/scripts/extract-vmlinux

> Also here is an output of `stap -vv`:
> 
> http://pastebin.com/raw/GEdMqL7U

OK, here we see pc=0xffffffff811fca70.  I downloaded your dbgsym
4.2.0-27.32 myself, and objdump gives me:

ffffffff811fca70 <SyS_open>:
ffffffff811fca70:       e8 cb 7a 5f 00          callq  ffffffff817f4540
<__fentry__>

This looks perfect.

But the closest kernel I could find is 4.2.0-27.32~14.04.1, and this
doesn't match at all:

ffffffff811fca6b:       48 c7 83 80 00 00 00    movq   $0x0,0x80(%rbx)
ffffffff811fca72:       00 00 00 00
ffffffff811fca76:       44 89 b3 88 00 00 00    mov    %r14d,0x88(%rbx)

ca70 would be probing in the middle of an instruction!

At least "readelf -n" on these two does show different Build ID values.

>> Are you sure the dbgsym you downloaded is an exact match for your
>> kernel?  It should have failed earlier if it wasn't, but a mismatch
>> could lead us to attempt kprobes on weird addresses, like in the
>> middle of an instruction.
> 
> `uname -r` says `4.2.0-27-generic` and installed packages are:
> 
> linux-headers-4.2.0-27
> linux-headers-4.2.0-27-generic
> linux-image-4.2.0-27-generic
> linux-image-4.2.0-27-generic-dbgsym
> linux-image-extra-4.2.0-27-generic
> 
> Looks right. Is it any other way to check this?

Try: dpkg --list 'linux-image*'

Ubuntu sometimes has sub-versions that don't show up in uname, like .32
in your dbgsym vs the .32~14.04.1 security update I found.  The running
kernel must match dbgsym exactly for us to do anything useful with it,
but like I mentioned we do runtime build-id checks which should catch
that before kprobes are attempted.

If that mismatch isn't being detected, that's a bug in itself!

I can't find exact dbgsym for the linux-image I downloaded.  If Ubuntu
doesn't publish this, you'll have to complain to them, or build and run
your own kernel with the necessary debuginfo.

>> Another thing to try is instead use "nd_syscall.open" etc. which will
>> probe functions by name, without requiring debuginfo at all.
> 
> Wow, it works! So... why I need debuginfo in the first place? Are there
> any features that will not work without it or something like this?

Without debuginfo, we can only use kprobes on the call and return of a
function by name; no probing inline functions or specific line numbers.
 We also don't get location info for any variables.  The nd_syscall
tapset has hard-coded the syscall parameter positions and reads them
according to normal calling ABI, but you can't rely on that everywhere.