This is the mail archive of the
systemtap@sources.redhat.com
mailing list for the systemtap project.
Re: Looking over LTT instrumentation
- From: Vara Prasad <prasadav at us dot ibm dot com>
- To: William Cohen <wcohen at redhat dot com>
- Cc: SystemTAP <systemtap at sources dot redhat dot com>
- Date: Wed, 02 Mar 2005 00:22:47 -0800
- Subject: Re: Looking over LTT instrumentation
- References: <4224F765.3010307@redhat.com>
William Cohen wrote:
This afternoon I looked over the LTT-0.9.6-pre4 kernel patch
from http://www.opersys.com/ltt/index.html. This was to see what event
and which locations in the kernel are insrumented, what kind of
functionality could be provided with the existing kprobe
instrumentation, what things need additional help to implement, and
what things are unlikely to be instrumentable. I will refer to things
by the LTT event define in most cases.
There are some LTT probe points that can be implemented with jprobes,
probe at function entry and just using arguments passed into function:
LTT_EV_PROCESS_WAIT
LTT_EV_TIMER_EXPIRED
LTT_EV_MEMORY_PAGE_FREE
LTT_EV_SOCKET_SEND
LTT_EV_SOCKET_RECEIVE
do_page_fault() entry
There are some that could implemented as regular function exit probes:
do_page_fault() exit
There are some that are basically exit probes but use an argument
passed into the function:
LTT_EV_SOCKET_CREATE (needs type arg)
LTT_EV_MEMORY_PAGE_ALLOC (needs order arg)
However, many LTT probes are the middle of functions, e.g. ones for
scheduling and exit logging. Commonly, there is some checking that the
operation can be done. After all the checks the operation is
guaranteed to be sucessful. The probe is inserted after those checks
to avoid writing a log entry for something that didn't actually
happen. Quite often there is a function call do the operation after
the checks, e.g. LTT_EV_MEMORY_SWAP_IN in
mm/filemap.c:wait_on_page_bit() immediately before swapin_readahead().
Maybe specify have kernel functions specify points where the error
checking is all done in the function and a probe could/should go
there, with a list of variables live there.
I am not sure i quiet followed the above statement. How do we specify the
location in the function after sanity checks, are you suggesting markers
of some kind.
Other alternatives spec of probe points:
1) have instrument callee functions and have predicates to specify
caller
This is a very interesting idea and i hope there are no caveats
in our ability to walk the stack and find our callers on all the platforms
reliably.
2) allow instrumentation to specify probe point before a call to
a particular function in the function.
Could do instruction after call, but need to be careful because
the following instruction could be in a different basic block
I like the idea of probe before a calling a function and i think we
can put a probe point reliably this way than after the call.
The do_traps logging in LTT will not work because kprobes uses traps
and it would be recursive.
LTT systemcall logging appears to collect do all or no
syscalls. Similarly for traps, all or nothing. Given that traps and
syscall do a wide variety of things, finer grain control of these
would be desired.
I agree with you that we need a fine grain granularity, we will get too
much data than needed with LTT all or nothing approach.
Need to get data transfer mechanism in place. printks are not going to
work well for the volume of data that LTT produces. However, it would
be hoped that some of the data analysis could be done in the probe and
reduce the need to produce a trace that is analyzed offline.
LTT uses relayfs and people have used this combination to collect very
high volume of data. Now that relayfs has gone through community
review we should definitely consider using it. Please let me know if you
need some help in putting together the transport for this work.
Wednesday I am going to see about implementing some of the LTT probe
points with kprobes.
Thanks for looking into this.
-Will