Re: breakpoint assistance (uprobes)

[Jim Keniston <jkenisto at us dot ibm dot com>]

On Fri, 2007-03-02 at 13:24 -0800, Roland McGrath wrote:
> I'm sorry for my long absence from the uprobes discussion.  Rather
> than trying to respond to all the individual questions and proposals,
> I'll just start in on the design topics that I think are most useful
> to pursue first.

Thanks for kicking off this discussion, Roland.  Unfortunately, I think
your email reached the systemtap list only yesterday.

> 
> The central component of the currently going uprobes ideas that needs
> to be worked out is what I'll call breakpoint assistance.  This is
> broadly useful for many utrace-based facilities to integrate, and any
> worthwhile implementation will become a modular component used by
> other things unrelated to systemtap.  Today I'll discuss the major
> design choices in this component.
> 
> On the status of utrace: it's now included in Andrew Morton's -mm tree;
> he tracks my current patchset from http://redhat.com/~roland/utrace/
> regularly.  The best way anyone can accelerate the process towards
> full upstream acceptance is to participate in the public review and
> discussion.  The correct forum for that is lkml (and CC: roland@redhat.com),
> not this mailing list.
> 
> Standard breakpoint insertion is straightforward and that is adequate.
> The complexity is in stepping over a breakpointed instruction to
> resume.  There are three essential implementation strategies available
> (leaving aside pure emulation, and I don't know of a fifth).
> 
> Stop all threads, remove breakpoint, single-step, re-insert, resume all.
> This is the traditional method used by debuggers.  It is reliable and
> machine-independent, but has always been slow.  Doing this via utrace in
> the kernel directly will be quite a bit faster than a user-level
> debugger can do.  This is the natural fallback position if other methods
> don't pan out.  The implementation of this is fairly straightfoward, but
> most of it will fall out of any new higher-level general facilities on
> top of utrace.  It's not especially interesting to do it independently
> in support of systemtap/uprobes.

Uprobes folks have talked a fair amount amongst ourselves about
single-stepping in multithreaded apps.  This approach was on our list,
but we haven't tried it because (1) it didn't seem very promising
performance-wise, and (2) it could really drive up the number of
SIGSTOPs utrace injects into the probed process.

> 
> Single-step out of line, in the style of kprobes.  I had been skeptical
> about doing this for user mode, but lately I'm feeling more optimistic.
> This appears to be the most promising direction in the short term.  
> Its implementation issues are complex.  I'll get into those in a
> separate thread momentarily.

Yeah, that's the approach we've bought into.

> 
> Fancy VM tricks.  This is the shangri-la path and the general subject
> has long been on my mind as future work for the utrace world.  The
> general capability required to do this is useful for a variety of other
> things, including thread-specific breakpoint insertion and a better way
> to do breakpoint insertion generally, and VM watchpoints.  I am full of
> ideas about this, but figuring it out requires some hard brainstorming
> with someone who knows the Linux VM code in much more detail than I do.
> I haven't had the chance to do that yet.  (What I have in mind also
> needs to wait for a little bit of fancier utrace infrastructure
> development, too.)  I don't know enough yet to be confident on the details
> of a design, but I know enough to be pretty sure that there is no simple
> "surgical strike" implementation plan that will work with acceptable
> properties for a production system.  I certainly think we'll pursue this
> in the long term regardless, but I think it's too big a nut to crack soon. 

This is not currently on our radar, at least as far as code probepoints
are concerned.

> 
> 
> Thanks,
> Roland

Jim