Re: Concurrent userspace probes targeting same location

[Wade Farnsworth <wade_farnsworth at mentor dot com>]

David Smith wrote:
> On 07/03/2012 03:06 PM, Wade Farnsworth wrote:
> > Greetings,
> >
> > I'm seeing some odd behavior when multiple userspace return probes are
> > running attached to the same target location.
> >
> > First, the return value of uaddr() of one probe points to a location
> > inside of the second.  The second probe's uaddr() behaves as expected.
> >
> > Second, it appears that occasionally a probe will get executed twice for
> > a specific function return.
> >
> > I have observed these problems on x86/Fedora 16, and ARM/Yocto.
> >
> > Here are some sample probes that exhibit the problems:
> >
> > probe process("/bin/ls").library("libc.so.6").function("malloc").return
> > {
> >      if (usymname(uaddr()) == "__libc_malloc")
> >          next;
> >      printf("malloc probe 1 -- ptr: 0x%x, return addr 0x%x\n", $return,
> > uaddr());
> > }
> >
> > probe process("/bin/ls").library("libc.so.6").function("malloc").return
> > {
> >      if (usymname(uaddr()) == "__libc_malloc")
> >          next;
> >      printf("malloc probe 2 -- ptr: 0x%x, return addr 0x%x\n", $return,
> > uaddr());
> > }
> >
> > And here is a snippet of the output when these are run simultaneously on
> > Fedora 16:
>
> When you say "run simultaneously", do you mean run as two separate
> scripts at the same time or run in the same script?

I encounter the problems in both situations.  Whether the probes are in 
separate scripts/stap instances or the same one, it doesn't seem to make 
a difference.

> > malloc probe 2 -- ptr: 0x9b12008, return addr 0xbfcbe010
> > malloc probe 2 -- ptr: 0x9b12008, return addr 0xbfcbe010
> > malloc probe 1 -- ptr: 0x9b12008, return addr 0x446f1901
> > malloc probe 2 -- ptr: 0x9b12020, return addr 0xbfcbe010
> > malloc probe 1 -- ptr: 0x9b12020, return addr 0x4469f1fc
> > malloc probe 2 -- ptr: 0x9b12030, return addr 0xbfcbe010
> > malloc probe 1 -- ptr: 0x9b12030, return addr 0x44698da2
> > malloc probe 2 -- ptr: 0x9b12020, return addr 0xbfcbe010
> > ...
> >
> > The lines prefixed with "malloc probe 2" consistently point into the
> > kernel module space (0xbfcbe010), instead of into userspace.  Also, the
> > first two lines demonstrate the double execution problem (note the
> > identical values of $return).
> >
> > Has anyone experienced anything similar?  Is this even a valid use case?
>
> Running multiple return probes at the same location in the same script
> should work.