Re: Fw: [ltc-interlock] [RFC] draft NFS trace hooks (fwd)

[Gerrit Huizenga <gh at us dot ibm dot com>]

Hi Tony - a few qiuck comments below in response to some of yours.

On Tue, 22 Aug 2006 11:43:20 +0200, Tony Reix wrote:
> Hi Li,
> 
> Le lundi 14 ao=FBt 2006 =E0 11:11 +0800, Li Guanglei a =E9crit :
> > Hi, Tony
> >
> ...
> 
> >   SystemTap provides an infrastructure to do a dynamic probing, which
> > means no patches to Kernel, no recompile and no reboot is required.
> 
> I think such features will be very useful in the field, when customers
> encounter critical problems and that stopping the machine or updating
> the kernel to a new version is not an option.
 
Yep - and that's why it was developed.  ;)

> >  And I attached a list of what we want to trace for NFS. But we are
> > not working on NFS itself and don't have much knowledge about it.
> 
> We are working since some years on NFSv4, mainly testing or adding IPv6.
> So we have some understanding of the protocol, but sure we do not master
> the protocol nor the code.

Think about this as well from the point of view of someone using or
debugging NFSv4.  That is the ultimate goal of systemtap - to aid the
system administrator or support person to find the *right* information
related to any subject area.

> >  I put some answers to your questions below.
> 
> > Please tell me if we missed some important functions that should be
> > probed for NFSv4. But what Xuepeng sent out is only our first step of
> > NFS trace hooks. The full list of our plan is in the attachment.
> 
> As I said before, we do not master NFSv4 internals, which are VERY
> complicated. I think only Trond Myklebust, Bruce Fields and Chuck Lever
> could really help you to check that these hooks are appropriate. But I
> know that Trond, Bruce and Chuck are very busy ... So they probably have
> no time to spend to understand SystemTap goals and basics and to check
> that the hooks you designed are appropriate.
> 
> Chuck asked for a design document. It seems you do not have one. And
> building such a document would require a deep knowledge of NFS/NFSv4
> internals. So this design document should be written either 1) by means
> of a collaborative work between SystemTap and NFSv4 experts, or 2) by
> someone who could spend time to build skills on both technologies, and
> then discuss with the SystemTap and NFSv4 experts.

Most of Linux and related utilities are not so lucky as to have
design documents, mostly because Linux evolves as do most open
source projects.

See Greg KH's slides - starting with the "The kernel has no obvious
design" slide.  It is pretty good, pretty interesting.

> > >  > I know very few about SystemTap. However, I have questions:
> > >  > - Do you have a design document describing what must be traced in NF=
> S ?
> >
> > See the attachment.
> 
> Hum. This looks more as a conclusion than as a design explaining the
> rationale of the choices.
> 
> 
> > >  > - How do you plan to submit your code to NFS or Kernel maintainers ?
> >
> > No. The trace hooks written in tapsets is for dynamic tracing, which
> > means that no patches to Kernel is needed.
> >=20=20=20
> > >  > - How much of your code must be included in NFS code, as patches ?=
> 
> >
> > None. This is the powerful aspect of SystemTap: you can probe the
> > Kernel without touching the Kernel source codes.
> >
> 
> Ohh. I did not know.
> Probably this is explained in the OLS'06 paper.
> Do you have a link to a technical explanation of how that works ?
 
Look up on, for instance, lwn the topic "kprobes".  There should be
lots of kprobes/dprobes information floating around which describes
the somewhat debugger like functionality of adding trace points.

> > >  > - Have you worked with them (I saw nothing on mailing list) ?
> >
> > No. we now work with SystemTap community. NFS is only a part of what
> > we are doing.
> 
> I do not remember someone talking about systemtap in the nfs and nfsv4
> mailing list. A quick search in the 3200 messages I've kept in my
> folders seems to show that no one talked about systemtap before you did.

Everything starts somewhere - consider this the first introduction.  ;)

> Do you know what is the position of OSDL about SystemTap ?

OSDL loves SystemTap.

> > >  > What are your plans about tapsets for NFSv4 ?
> >
> > only NFSv4 procedure stub functions for the client and server side.
> >
> > >  > Do you have resources to do that in 2006 or in 2007 ?
> >
> > oh. I am not sure. We are assigned to work for SystemTap and LKET :-)
> 
> I'm used to have students giving help on some projects (this year: NPTL
> Trace Tool, and NFSv4 Administration/Security).
> Do you think a student (best French Computer Science University, 5-6
> months of work) could help ?
 
Possibly - although another project might be to help write tapsets for
various areas of the kernel.  It is a good way for students to become
more familiar with specific areas of the kernel that might be interesting.

> > >  > First step could be to check that NFSv4 developers are already Syste=
> mTap
> > >  > enthusiastic. If not yet, one should discuss of that with them. It s=
> eems
> > >  > very important to get their comments and approval for which features=
>  to
> > >  > trace. The problem is that NFSv4 Server developer is overloaded ... =
> so
> > >  > it may take some time.
> > >  > Also, there already are some trace code in NFSv4.
> >
> > It seems to me that a lot of developers still not realized the
> > existence or how SystemTap could facilitate their
> > development/debugging of Kernel. We need more advertisement for
> > SystemTap :-)
> 
> Yes. I've talked with 2 guys here working on ext3 and they have no clear
> idea of how SystemTap could help them ... One of them (a ext3 and Xen
> expert) did not know about SystemTap even.

I'll help educate your local team on SystemTap soon, it's benefits and
our general goals.

> I know that proposing debugging/tracing tools to expert/guru developers
> is difficult: they are often more efficient without using such tools but
> they often forget that such tools will greatly help maintenance in the
> future and to understand problems in the field.
> 
> What do Linux gurus (Linus, ...) think about SystemTap ?
> Now, it seems that SystemTap is only an initiative by RedHat, IBM, Intel
> and Hitachi. People close to the customers.
 
SystemTap is getting a good rep in the customer field for sure and if
that were all it ever got, it would still be worthwhile.  However,
Dave Jones referenced it as well in his talk at OLS: 

http://www.linuxsymposium.org/2006/view_abstract.php?content_key=193
http://kernelslacker.livejournal.com/44060.html

Dave points out that SystemTap could have made his kernel level
analysis much simpler, as one example.

> What about a student working on tapsets for NFSv4 in 2007 ? After we
> have got the position of NFSv4 developers about SystemTap, sure. I'll
> ask them about SystemTap.

Yep - that would be very cool!

gerrit