Re: process().statement() doesn't seem to work

[Josh Stone <jistone at redhat dot com>]

On 06/26/2015 11:58 AM, Zoltan Kiss wrote:
> Hi,
> 
> I want to use systemtap to figure out the local variables in a function 
> where something goes fishy. The function's code is here:
> 
> http://dpdk.org/browse/dpdk/tree/lib/librte_pmd_ixgbe/ixgbe_rxtx.c?id=v1.7.1#n1313
> 
> It's a receive function of DPDK's userspace poll mode driver, so it's 
> called in an infinite loop. When I try this, it appers to work:
> 
> probe 
> process("/usr/sbin/ovs-vswitchd").function("ixgbe_recv_scattered_pkts").return 
> {
>    log($$locals)
>    exit();
> }
> 
> The output is:
> 
> rxq=0x7fff852ee000 rx_ring=? rxdp=? sw_ring=? rxe=? first_seg=? 
> last_seg=? rxm=? nmb=? rxd={...} dma=? staterr=? hlen_type_rss=? rx_id=? 
> nb_rx=0x0 nb_hold=0x0 data_len=? pkt_flags=?
> 
> But it doesn't show most of the variables I need. It returns nb_rx=0, so 
> I know where are the two points where things can go wrong, but I can't 
> see the output of those variables (staterr and nmb)

The behavior of .return variables is often suprising to people.  By the
time a .return probe is reached, the program actually will have already
left that stack frame.  Technically, that means only the $return value
is available to read.

But we cheat a little and also save values from the *entry* of the
function for you.  That's most useful for the $$parms string or any of
the individual parameters.

Sometimes you can also get a glimpse of $$locals, if the debuginfo tells
us how, but again this is only from the very beginning of the function.
 In your case, the three visible values all happen to have trivial
initialization: nb_rx = 0; nb_hold = 0; rxq = rx_queue;

So you can't trust that these were still the values after the function
returned.  For that, you really need to probe the statement just before
it actually returns.  Even on the return statement itself is ok.  Since
your function has just one return, it looks like line 1574 is best.
(but depending on optimizations, some locals might be missing there.)

> When I try to define the probe as this:
> 
> probe 
> process("/usr/sbin/ovs-vswitchd").statement("*@lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1359")
> 
> Or with any line number inside that function, the probe is never 
> reached. I'm using gcc 4.8.4, DPDK is statically linked to my application.
> Does anyone have an idea what might went wrong?

It may be that you're really not reaching that line, e.g. if nb_pkts==0.

You can see all the lines that stap can probe like this:

stap -l 'process("/usr/sbin/ovs-vswitchd")
         .statement("ixgbe_recv_scattered_pkts@*:*")'

Or change -l to -L to also show all readable variables at each point.


You might like the varwatch example script:

https://sourceware.org/systemtap/examples/#general/varwatch.stp

It takes two arguments - first a probe pattern like your
process.statement wildcard, then the variable to watch like $$parms,
$$locals, or $$vars for both.  A single name like $nb_rx is fine too.