Looking over LTT instrumentation

[William Cohen <wcohen at redhat dot com>]

This afternoon I looked over the LTT-0.9.6-pre4 kernel patch
from http://www.opersys.com/ltt/index.html.  This was to see what event
and which locations in the kernel are insrumented, what kind of
functionality could be provided with the existing kprobe
instrumentation, what things need additional help to implement, and
what things are unlikely to be instrumentable.  I will refer to things
by the LTT event define in most cases.

There are some LTT probe points that can be implemented with jprobes,
probe at function entry and just using arguments passed into function:

	LTT_EV_PROCESS_WAIT
	LTT_EV_TIMER_EXPIRED
	LTT_EV_MEMORY_PAGE_FREE
	LTT_EV_SOCKET_SEND
	LTT_EV_SOCKET_RECEIVE
	do_page_fault() entry

There are some that could implemented as regular function exit probes:
	do_page_fault() exit

There are some that are basically exit probes but use an argument
passed into the function:
	LTT_EV_SOCKET_CREATE (needs type arg)
	LTT_EV_MEMORY_PAGE_ALLOC (needs order arg)


However, many LTT probes are the middle of functions, e.g. ones for
scheduling and exit logging. Commonly, there is some checking that the
operation can be done. After all the checks the operation is
guaranteed to be sucessful. The probe is inserted after those checks
to avoid writing a log entry for something that didn't actually
happen. Quite often there is a function call do the operation after
the checks, e.g. LTT_EV_MEMORY_SWAP_IN in
mm/filemap.c:wait_on_page_bit() immediately before swapin_readahead().
Maybe specify have kernel functions specify points where the error
checking is all done in the function and a probe could/should go
there, with a list of variables live there.

Other alternatives spec of probe points:
1) have instrument callee functions and have predicates to specify
	caller
2) allow instrumentation to specify probe point before a call to
	a particular function in the function.
	Could do instruction after call, but need to be careful because
	the following instruction could be in a different basic block


The do_traps logging in LTT will not work because kprobes uses traps
and it would be recursive.

LTT systemcall logging appears to collect do all or no
syscalls. Similarly for traps, all or nothing.  Given that traps and
syscall do a wide variety of things, finer grain control of these
would be desired.

Need to get data transfer mechanism in place. printks are not going to
work well for the volume of data that LTT produces. However, it would
be hoped that some of the data analysis could be done in the probe and
reduce the need to produce a trace that is analyzed offline.

Wednesday I am going to see about implementing some of the LTT probe
points with kprobes.


-Will