Re: corrected: tapset/script information template

[William Cohen <wcohen at redhat dot com>]

Frank Ch. Eigler wrote:
> Hi -
>
> wcohen wrote:
>
>
> > I have attempted to write up a description for dynamic callgraph 
> > instrumentation based on the template.
>
>
> Excellent, thank you.  I committed a copy to CVS (under src/tapsets)
>
>
> > [...]
> >  $dynamic_call_graph = 1; // turn on tracing of calls for thread
> >  $dynamic_call_graph = 0; // turn off tracing of calls for thread
>
>
> Do you imagine that just setting those variables in some other probe
> handler should activate this kind of tracing?  It could be partially
> implemented using a script-level construct like this (also committed):
>
> global $dynamic_call_graph
> probe kernel.perfctr.call(1) { 
>    if ($dynamic_call_graph) trace_sym ($pc);
> }
> probe kernel.perfctr.return(1) { 
>    if ($dynamic_call_graph) trace_sym ($pc);
> }
>
> Boy, that looks *expensive*.
>
>
> This reminds me of an oprofile-related issue we've probably discussed
> before: whether these events should be signalled with the NMI versus
> an ordinary interrupt.  We'd have to prevent reentrancy (nested probe
> handler execution) somehow.  With an NMI, it'd have to be explicit
> code that tracks the current processor's handler-active state,
> dropping the event with an early-return.  With a maskable interrupt,
> it could defer the hit until interrupts get unmasked, at which point
> the collectable data may not be accurate.

I was thinking there are very specific places that one would want to 
start tracing and that it would be controlled by some probes, for 
example turning on the dynamic call graph when entering a "function of 
interest"  and turning it off when exiting the same function.

I was thinking that instrumentation would be assisted by the debug 
hardware rather than the performance monitoring hardware. The problems 
with using the oprofile style support is that the sampling is imprecise 
and very machine specific. There is precise event sampling buffer on the 
P4, but the debug branch tracing mechanism in the p6 and newer 
processors looks a bit more portable than the p4 only performance 
monitoring hardware.  The debug also looks a bit easier to use: fewer 
bits to twiddle and easier to find the from/to address.

However, the using debug hardware is still going to be expensive. It 
doesn't distinguish between braches, calls, or returns, so for each 
branch taken the trap handler is going to need to examine the 
instruction to determine whether it cares about it. One upside is that 
would only affect the thread being traced. The perfmon approach would 
probably affect everything.

The NMI vs IRQ for sampling is an issue. Some people really like the NMI 
sampling of oprofile because it can show what is going on inside the irq 
masked regions.

For the dynamic call graph tapset, the data written would be pretty 
limited. However, I could see where it would be possible to construct 
something where the dynamic call graph tapset function call something 
that triggers another probe.

-Will