Re: user-space probes -- plan B from outer space

[Vara Prasad <prasadav at us dot ibm dot com>]

Frank Ch. Eigler wrote:

> Hi -
>
> Here is an outline of how systemtap might support user-space probes,
> even in the absence of kernel-based user kprobes.  This is a "plan B"
> only, a desperate stopgap until lkml sees the light.  Maybe "plan Z"
> is more appropriate, considering the limitations I'm about to outline.
>
> The idea is to support limited systemtap scripts that refer only to
> user-space probe targets such as existing processes.  These scripts
> would be translated to a user-space probe program instead of a kernel
> probe module.
>  
Well another  approach is similar to Ptrace actions you have predefined 
handlers that some one can activate through a new systemcall. This gives 
the ability execute simple handlers without a need for loading kernel 
modules. For those who wants to do complicated aggregations, correlation 
etc., you can load your own handlers through a module and activate them 
through the same systemcall.  This gives the flexibility for power users 
and simplicity for regular users and we can get away from module 
loading/root permission issues for simple tracing.

> Probes would be specified with a probe point syntax such as:
>
>   user.process(233).statement(0xfeedface)
>   user("fche").process("/bin/vi").function("*init*")
>
>  
In my opinion user space probes is most useful in the case of server 
class kind of complicated programs which are usually long living, with 
that background i feel there is not much value with system wide tracing 
instead we should focus on process specific tracing. We could also allow 
users to start a program using systemtap so that we have a chance to 
place probes before running the process, this could be useful to track 
problems during the startup, and of course in this case no pid will be 
specified. 
I am not sure i see the value of process("process name") syntax if our 
focus is process specific tracing.

> Instead of kprobes of a probe module, this probe program would use
> ptrace to insert breakpoints into any target processes, perhaps using
> code from RDA or GDB.  Given the process-id or process name, systemtap
> should be able to locate the necessary debugging information at
> translation time.  When probes are hit, the probe process would run
> the compiled probe handlers in much the same way as now.  Access to
> $target vars should be possible.  The runtime code would have to have
> a new variant to use some user-level facility (plain pipes?)  to
> communicate with the front-end.
>  
I am not sure i see lot of value of this solution compared to a gdb 
batch job, but for bit better performance than the heavy weight gdb.
I keep receiving complains from people that strace performance overhead 
makes it prohibitive to solve performance related problems and this is 
going to be even more slow than that so we have to keep that in mind as 
well.

> Q: Wouldn't this be slow?
> A: Oh yes, quite.  Several ptrace context-switch round-trips per
>   probe hit.  Lots more if we want to pull out target-side
>   state like $variables or stack backtraces.
>  
Jim last week did a simple performance comparison of current uprobes 
solution vs strace solutions to track a process his results were strace 
was 16x slower than current in kernel handlers.

> Q: What about concurrency?
> A: You mean like probes concurrently hit in several target processes,
>   like SMP kprobes?  If there was any indication that this was
>   worthwhile, then we could make the systemtap-generated probe
>   process be multi-threaded (one probe thread per target thread).
>  

I would say it is probably not worth the complication instead one 
systemtap process can trace only one user process, so if one wants to 
track several process from several windows run several sessions of  
systemtap, i think this may be a good start than making complicated 
multithreaded stap.

> Q: Any other limitations?
> A: Because of ptrace, any process can be supervised by only one
>   process at a time.  So if you run systemtap on a user process,
>   you won't be able to run gdb or another systemtap session on it.
>
> Q: What about probing the kernel and user space together?
> A: Maybe this scheme would work if kernel-space systemtap probes
>   run concurrently, and arrange to share systemtap globals with
>   userspace somehow (mmap?).  Shared variables like this would
>   likely cause many more locking timeouts (=> skipped probes)
>   than now.  There are also additional security concerns.
>  
We could introduce a new systemcall that can be called from the handler 
to execute a kernel handler to get correlated info but as you said other 
security concerns might not make it possible to get this accepted.

> Q: What about probing shared libraries?
> A: Because of the way ptrace works, we'd have to turn these into
>   process-level probes, including probes that just sit around
>   monitoring the threads and all their children to dlopen/mmap
>   the named libraries.
>
>  
Well if we just admit that we allow process based probing we dont need 
to do any thing more for shared libraries specifically.

> Q: Is it worth it to try?  Is there a better way?
> A: You tell me.
>
>  
Not really unless we can't come up with a simple enough of solution that 
lets handlers run in the kernel which gives the performance that we need 
even then we have to justify what is the value of this duplication of 
gdb functionality.

> - FChE
>