Minutes of last 3 weeks meetings

[Vara Prasad <prasadav at us dot ibm dot com>]

Hi,

I am sending the minutes of our last three weeks meetings.
I am sure i have not covered all the items in my minutes that we
talked. Please feel free to correct and add what is missing.

Thanks,
Vara Prasad

--------------------------------------------------------------------------------------
2/24/05
--------
Attendees: Vara, Hien, Will, Frank,  Brad, Sri and Richard

Richard said there are two different goals namely Performance goals and 
Debugging goals
one needs to be aware of both for this project.
Some times The goals of the performance and debugging may be at odds.

Based on Will's work Kprobes have an overhead of 1 MS.
This was the same overhead that Richard found when
he did this expt. long back on Pentium 90Mhz.
Jprobes on AMD is 300 cycles faster.
We discussed if we need to support "*" wild card syntax in the language
and the consensus is yes.
We discussed a need to define co-oridnate system to identify the location of
specifying probes. Line number is not the good one. Vara took an action
item to start the definition process.
We have to consider not synchronous execution related events but also
asynchronous events related to timer, cache misses etc. in the definition.
Vara mentioned that recently in a customer site Solaris folks were able 
to analyze
performance bottlenecks in minutes where as in Linux they couldn't find 
it easily due to
lack of tools. Sridhar from Intel wanted to know the specifics of the 
problem and
Vara has promised to give it to him.

3/3/05
Jeff, Sri, Brad, Will, Martin, Frank, Tom, Jim, Hien and Vara, Larry

Vara asked if we should collaborate with LTT  and LKST teams the answer
was yes. Vara has an action item to initate a contact with LKST team.
Brad said Intel would be interested in contributing to Parallel 
programming TAP
sets like lock and scheduler from Intel.
Will asked if Intel would be interested in Virtualization issues.
Brad said yes.
We need to think about both kernel debugging and performance
issues. Brad said Performance experts may not have the know how of
the kernel but they would like to analyze the data from
a performance point of view.


3/10/05
Jeff, Elena, Ulrich, Sri, Ananth, Will, Martin, Jim, Hien, Frank, Larry

Agenda:
Discussion about ways to define a probe
Discussion about exit probes
Discussion about run time
Discussion about other studies Will did.

This call was mostly to go over Vara's probe definition email and decide
which ones we should implement and which we should not.

goto labels may be not be preserved needs investigation
Jim to do the expt and post the results.
Jim already posted the results and found that labels are
not reliable based on the compiler optimization.
Line numbers are pretty dangerous with compiler optimizations.
Frank: thinks we need to support line numbers and he thinks
changing version numbers wont cause panics.
Ulrich doesn't agree with that view. He said compiler
optimizations can cause problems but may not panic.
In spite  of all the issues related to line number it is the
most logical way for programmers to think hence we concluded that
we will support. We might want to support only in power user mode
rather than a general user mode. 
Sridhar from intel asked how do we insure that it is a power user
answer Vara gave was system wont ensure but users have to be
aware of this. Distro's can package it in such a fashion that these
features are not builtin to production systems.
Ulrich said we can not differentiate between multiple function
calls to the same function hence when someone specifies a probe
before a function it should be every function call.
Item 6 can be implemented but not scalable for frequently called functions.
Item 6 and 7 can be implemented as one.
probe definition has to be thought in terms of line and
column numbers.
Ulrich said gcc can be changed to emit the column info.
at lower debug levels than -g3.

NO OP markers can have significant performance impact on
platforms like IA64. We have to be careful in which
part of the kernel these markers are used.
These markers can be made such that they can be compiled out
when not needed. NOOP markers make compiler generated code
different due to memory barriers. Sun might claim that NOOP
markers themselves may not add any significant overhead but they
may not account for the performance difference in the code
generated with and without NOOP's is what Ulrich mentioned.

Second type of static trace we only have simple global flag
check then jump to another portion of the code to do the
tracing. This will decrease the code blot.

Will suggesting to define probes based on data structure
mainly function pointers.
Ulrich said what need for this is a facility to instrument functions named
in the data structures with the given  labels.