Re: [UPDATE] Linux Kernel Event Trace tool(LKET)

[Li Guanglei <guanglei at cn dot ibm dot com>]

Stone, Joshua I ??:
> Hi Li,
>
> Currently I am working on a few generic tapsets, and there appears to be
> some overlap with what you've done.  For example, earlier this week I
> checked in a process tapset.  This should be able to give you the fork
> and exec events you're using in LKET.
>
> Thang and I are also working on IO, scheduler, networking, and VM
> tapsets, but we don't have anything ready to check in yet.  I will
> definitely take a look at what you've provided though, and leverage that
> which overlaps.
>
> I think it would be great to have LKET as a tapset, but I would like to
> see a more layered approach.  What you've given is very specific to
> tracing, but the events that you probe have a more general audience as
> well.  A broader approach would be to implement the events as general
> tapsets, and then build LKET as a layer on top of the base tapsets.
>
> The syscalls are a good example of this -- in LKET you've captured
> syscalls using function wildcards.  However, we have the syscall tapset
> that has already gone through pains to define all of the actual system
> calls available.  If we run into syscall issues in the future, it would
> be much nicer to need only fix the syscall tapset, and let LKET benefit
> from that.  So instead of probing 'kernel.function("sys_*")', you should
> use 'syscall.*'.  And if there are reasons why this doesn't work well
> for you, let's address that rather than duplicating efforts.
>
>
> Josh

So I'd like to break LKET's tapsets into the following two layers:

<1> Generic Tapsets
The first layer are generic tapsets which could not only be used for 
tracing, but also for general purpose diagnosis. It will include the 
probe points defined in current LKET's tapsets and some stap script 
statements of getting the interesting data related to that event.

So this will need some .stp files to be added into 
/usr/local/share/SystemTap/tapsets. These files are scsi.stp, 
ioscheduler.stp, tskdispatching.stp, networking.stp etc. Take an 
example of scsi.stp, it will look like:

...
probe scsi.ioentry
        = module("*").function("scsi_prep_fn@drivers/scsi/scsi_lib.c")
{
    disk_major = $rq->rq_disk->major
    disk_minor = $rq->rq_disk->>first_minor
    dev_state =  $q->queuedata->sdev_state
    // maybe a argstr?
}
...

These stp scripts not only address where inside the kernel to be 
instrumented, but also get out the interesting data related to that 
probed points. They look in the same style as current syscall.stp 
shipped with SystemTap.

<2> Tapsets Specific to Tracing
The second part will be tracing specific, which may be located in 
/usr/local/share/systemtap/tapsets/LKET/. They include those embedded 
c functions in current LKET. Take an example of scsi.stp:

probe addevent.scsi.ioentry
	= probe scsi.ioentry
{
	log_scsi_ioentry(HOOKID_SCSI_IOENTRY, $q, $req, backtrace)
}

Thanks to the optimizations of translator, the statements of 
"disk_major = $rq->rq_disk->major" will be compiled away and the 
generated c file will look exactly like the original LKET.

And "stap -bM -I /usr/local/share/SystemTap/tapsets/LKET sample.stp" 
will work the same as current LKET.

I'd like to check into CVS the first part of generic tapsets as a 
start point of getting more event hooks into SystemTap. And then 
checking the tracing specific tapsets into SystemTap's tapset library.

Any comments about this?

--Guanglei