This is the mail archive of the systemtap@sources.redhat.com mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Looking over LTT instrumentation


William Cohen wrote:

This afternoon I looked over the LTT-0.9.6-pre4 kernel patch
from http://www.opersys.com/ltt/index.html.  This was to see what event
and which locations in the kernel are insrumented, what kind of
functionality could be provided with the existing kprobe
instrumentation, what things need additional help to implement, and
what things are unlikely to be instrumentable.  I will refer to things
by the LTT event define in most cases.

There are some LTT probe points that can be implemented with jprobes,
probe at function entry and just using arguments passed into function:

    LTT_EV_PROCESS_WAIT
    LTT_EV_TIMER_EXPIRED
    LTT_EV_MEMORY_PAGE_FREE
    LTT_EV_SOCKET_SEND
    LTT_EV_SOCKET_RECEIVE
    do_page_fault() entry

There are some that could implemented as regular function exit probes:
    do_page_fault() exit

There are some that are basically exit probes but use an argument
passed into the function:
    LTT_EV_SOCKET_CREATE (needs type arg)
    LTT_EV_MEMORY_PAGE_ALLOC (needs order arg)


However, many LTT probes are the middle of functions, e.g. ones for scheduling and exit logging. Commonly, there is some checking that the operation can be done. After all the checks the operation is guaranteed to be sucessful. The probe is inserted after those checks to avoid writing a log entry for something that didn't actually happen. Quite often there is a function call do the operation after the checks, e.g. LTT_EV_MEMORY_SWAP_IN in mm/filemap.c:wait_on_page_bit() immediately before swapin_readahead(). Maybe specify have kernel functions specify points where the error checking is all done in the function and a probe could/should go there, with a list of variables live there.

I am not sure i quiet followed the above statement. How do we specify the location in the function after sanity checks, are you suggesting markers of some kind.


Other alternatives spec of probe points: 1) have instrument callee functions and have predicates to specify caller

This is a very interesting idea and i hope there are no caveats in our ability to walk the stack and find our callers on all the platforms reliably.

2) allow instrumentation to specify probe point before a call to
    a particular function in the function.

    Could do instruction after call, but need to be careful because
    the following instruction could be in a different basic block

I like the idea of probe before a calling a function and i think we can put a probe point reliably this way than after the call.



The do_traps logging in LTT will not work because kprobes uses traps
and it would be recursive.

LTT systemcall logging appears to collect do all or no
syscalls. Similarly for traps, all or nothing.  Given that traps and
syscall do a wide variety of things, finer grain control of these
would be desired.

I agree with you that we need a fine grain granularity, we will get too much data than needed with LTT all or nothing approach.


Need to get data transfer mechanism in place. printks are not going to work well for the volume of data that LTT produces. However, it would be hoped that some of the data analysis could be done in the probe and reduce the need to produce a trace that is analyzed offline.

LTT uses relayfs and people have used this combination to collect very high volume of data. Now that relayfs has gone through community review we should definitely consider using it. Please let me know if you need some help in putting together the transport for this work.


Wednesday I am going to see about implementing some of the LTT probe points with kprobes.

Thanks for looking into this.


-Will




Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]