user-space probing and fast kprobes

[Mathieu Lacage <Mathieu dot Lacage at sophia dot inria dot fr>]

hi,

I have tried to learn as much as possible from the current cvs source
tree, the existing documentation and the kprobes code and documentation
but there are still a few things I have not been able to figure out on
the two things I am really interested in: "user-space probes" and "fast
kprobes". 

As a developer, I get to debug multithreaded multimedia and networked
applications on mono-processor systems. Typically, I like to be able to
monitor kernel-level events with user-space events and I like to be able
to relate these different types of event with each other (so you can
tell if the kernel scheduled to thread X before getting the IO end event
or the PCI interrupt or a userspace function was executed):
  - kernel scheduling decisions
  - page faults for userspace programs
  - IO start, blocking and end events
  - special interrupts from a special PCI device.
  - userspace events such as: "Function X in thread Y of pid Z entered".

Another scenario of monitoring I would like to be able to do (although I
don't do that professionally very often) is be able to monitor a GUI
application running on X with proper monitoring of the events within the
X server, within the kernel DRI/DRM drivers, within the X graphic
drivers (plain 2D and openGL), within the window manager, the
compositing manager and the application.

For a long time, I have been using adhoc tools, I even ended up using
LTT in a few really bad situations and I must say that I am not very
satisfied with what I have been doing: in most cases, I had to write
adhoc code to be able to relate user-space events with kernel-space
events and that proved to be often quite painful. For LTT, I used a
special kernel syscall to notify the kernel-level tracing tool of a
userspace event: this allowed me to precisely be able to see the order
of events. 

Of course, in this context, systemtap looks like the right tool for the
job and the functionality I am most interested in is:
  - user-level probing: to be able to relate my user-space events with
the kernel-space events.
  - fast probing: to be able to handle high-frequency events. A recent
thread seems to show that I am not the only one considering this as an
important issue.

As of today, it looks like none of these two issues are being worked on
so I wanted to try to state a bit more clearly my requirements as a user
in the hope that I get heard by the developers. 

I am also interested in working on some of these issues in my spare time
so I would like to know if there is a consensus on how to address these
issues. 

As far as I can tell, it should be possible to deal with user-level
probing by using a kernel thread to ptrace the target PID and either
insert arbitrary int3 operations in the process address space or
manipulate the target process debugging registers. Of course, that would
mean:
  - raise interrupt
  - transform interrupt in signal
  - deliver signal to ptracing process
  - ptracing process handles signal
  - ptracing process stops signal delivery to ptraced process
If int3 is used (when the debugging registers are exhausted for
example), it would mean also the use of the step-by-step debugging
register bit to continue execution of the process without disabling the
breakpoint which means that for each breakpoint, you will need to go
through the process described above twice: once for the initial int3,
and once for the step-by-step interrupt after disabling the int3 to re-
enable it.

Do you think this approach would work ? If so, it is quite likely that
it would not fit the bill of the "fast probing" requirement I put forth
earlier. In that case, are there other implementation strategies which
are worth being considered ?

Mathieu
--