Re: cost of kprobe and jprobe operations

[Maneesh Soni <maneesh at in dot ibm dot com>]

On Tue, Feb 22, 2005 at 02:16:38PM -0500, William Cohen wrote:
<snip>
> 156      11.2069  memcpy
> 142      10.2011  restore_all
> 127       9.1236  _spin_lock
> 119       8.5489  kprobe_handler
> 110       7.9023  do_debug
> 105       7.5431  kprobe_exceptions_notify
> 93        6.6810  setjmp_pre_handler
> 90        6.4655  longjmp_break_handler
> 77        5.5316  __module_text_address
> 58        4.1667  resume_execution
> 51        3.6638  _spin_unlock
> 43        3.0891  notifier_call_chain
> 39        2.8017  int3
> 37        2.6580  get_kprobe
> 26        1.8678  do_int3
> 22        1.5805  debug_stack_correct
> 17        1.2213  debug
> 12        0.8621  _spin_lock_irqsave
> 9         0.6466  lock_kprobes
> 4         0.2874  _spin_unlock_irqrestore
> 4         0.2874  poison_obj
> 4         0.2874  unlock_kprobes
> 
> -Will



On Tue, Feb 22, 2005 at 05:20:17PM -0500, William Cohen wrote:
> Frank Ch. Eigler wrote:
> >Hi -
> >
> >
> >
> >>[...]
> >>These performance discussions might be more meaningful if 
> >>we had explicit performance goals for systemtap. [...]
> >>Any thoughts on defining performance goals for systemtap?
> >
> >
> >I don't think picking numbers does harm, but I don't know how
> >much good that does either.  Would we stop-ship a systemtap
> >implementation that didn't meet the goals?  In the abstract,
> >should one opt to optimize vs. extend functionality?  Maybe
> >these questions should be put aside until something runnable
> >exists as a point of reference, when actual decisions might
> >need to be made.
> >
> >
> >- FChE
> 
> I generated these number to give some concrete numbers on the cost of 
> the operations. We do want to be aware of the costs of the operations. 
> There are certainly going to be some points where instrumentation is 
> going to cause excessive overhead.
> 
> There are other performance issues with kprobes. There is a single 
> spinlock used by all kprobes and this is going to serialize the probes. 
> This will have a impact on smp scalability.
> 
> -Will


Very interesting numbers.. As earlier mentioned by Prasanna, I think 
we can use RCU to reduce the lock overhead. The golbal spinlock is for 
kprobes hash table and it is read-mostly data structure. As of now I think 
we can use RCU in this case. I will try to come up with a patch using RCU 
for kprobes hashtable. Hopefully that will improve the numbers.

Thanks
Maneesh


-- 
Maneesh Soni
Linux Technology Center, 
IBM India Software Labs,
Bangalore, India
email: maneesh@in.ibm.com
Phone: 91-80-25044990