Re: utrace-based uprobes

[Jim Keniston <jkenisto at us dot ibm dot com>]

On Fri, 2007-03-16 at 17:28 +0100, Mark Wielaard wrote:
> Hi,
> 
> On Thu, 2007-03-15 at 17:21 -0700, Jim Keniston wrote:
> > o As of now, uprobes aren't inherited across fork()s.
> 
> Is there a special reason for this? It looks like you currently do extra
> work in case of a fork() to remove the breakpoints, which most likely
> will be cleared anyway as soon as the thread does an exec().

The main reason is that a u[ret]probe object can be associated with only
one process at a time.  We could clone the u[ret]probe objects for the
new process, but it's not clear how those new objects' addresses would
be communicated back to the instrumentation module.  I'm sure it's
doable; it's just not high on our list.  Suggestions welcome.

> 
> > o Probe registration and ungregistration in the context of a
> > multithreaded application, are asynchronous events. All threads
> > need to be QUIESCED before the program text is modified to insert
> > the breakpoint.
> 
> Is this a requirement? I suppose modifying code while another thread can
> execute that code path is never safe. But do you have to suspend all
> threads? A suggestion would be to mark the code page as not executable
> so any thread that accesses that code path gets trapped till the code
> has been adjusted.

We settled on the quiesce-everybody approach a long time ago.  If we
rejected your idea, I can't remember why.  Maybe a colleague will inform
me.  Otherwise, it seems worth investigating.

> 
> > +When a CPU hits the breakpoint instruction, a trap occurs, the CPU's
> > +user-mode registers are saved, and a SIGTRAP signal is generated.
> > +Uprobes intercepts the SIGTRAP and finds the associated uprobe.
> 
> One problem that I recently encountered is that the process might have
> SIGTRAP masked or you might insert and then hit a breakpoint while a
> SIGTRAP handler is running in the user process (which normally means the
> handler is masked and hitting the breakpoint will then most likely kill
> the process completely). I don't know a way around this. If you can
> detect such a case you would need to enter the kernel differently. On
> the other hand this might not be a very common case. But it might be
> something you want to keep in mind.

OK.  At the very least, we'd need to document this as a limitation.

> 
> > +If the number of times a function is called does not match the
> > +number of times it returns (e.g., if a function exits via longjmp()),
> > +registering a return probe on that function may produce undesirable
> > +results.
> 
> What do these undesirable results include

Returning to the wrong address.  For example, if A calls B, and you have
uretprobes on both A and B, and B longjmps back into A, B's return
address will be left atop that task's stack of return addresses.  So
when A returns, it'll return using B's return address.

> (is it unsafe to add return
> probes)?
It's safe within the stated limitations.  In particular, it's fine if a
process exits from within a uretprobed function.

> Is there a way to detect such functions/behavior?
Perhaps the uretprobe function-entry handler could remember the stack
pointer, and the trampoline (return) handler could perform an
architecture-specific verification that the stack pointer points where
it should.

> 
> > +When you register the first probe at probepoint or unregister the
> > +last probe probe at a probepoint, Uprobes asks Utrace to "quiesce"
> > +the probed process so that Uprobes can insert or remove the breakpoint
> > +instruction.  If the process is not already stopped, Utrace sends it
> > +a SIGSTOP.  If the process is running an interruptible system call,
> > +this may cause the system call to finish early or fail with EINTR.
> 
> Is this a limitation of utrace at the moment? Or will it be possible in
> the future for utrace to "quiesce" a process without a "user visible"
> SIGSTOP signal?

That's one for Roland.
> 
> Cheers,
> 
> Mark

Good ideas.  Thanks.
Jim