Re: static instrumentation for kernel

[Mathieu Desnoyers <compudj at krystal dot dyndns dot org>]

Hi,

Just so you remember me, I was in LTT team at the OLS presentation when we first
discussed about these "markers". I am presently working on the next generation
of LTT (the LTTng tracer and LTTV viewer). As I speak, there is already a
feature-incomplete version of the tool ready (see http://ltt.polymtl.ca).


* Frank Ch. Eigler (fche@redhat.com) wrote:
> The code to be inserted into kernel sources would be a plain macro
> call such as:
> 
>    SYSTEMTAP_PROBE(name) 
>    SYSTEMTAP_PROBE_N(name,arg1) // arg1 castable to int64_t numeric
>    SYSTEMTAP_PROBE_NS(name,arg1,arg2) // arg2 castable to char* string
>

It seems to somehow limit the variations of parameters that can be passed as
argument : one could need a particular macro for a (int64_t,void*,int32_t,byte).
We can imagine thousands of variations. Or maybe the goal is just to allow a
small subset of types ?

> The name should be unique within the function.  As you see, arguments
> can be passed, encoding the type/arity into the macro name.  Possibly
> some super clever typeof() conditionals can make that implicit.

The problem with the precompiler is that it's not very "variable argument list"
friendly. It only passes it as __VA_ARGS__ to another function call, but doesn't
play with them.

A solution for using typeof() here would be to do something that looks like
system calls :

SYSTEMTAP_PROBE(name)
SYSTEMTAP_PROBE1(name,arg1)
SYSTEMTAP_PROBE2(name,arg1,arg2)
SYSTEMTAP_PROBE3(name,arg1,arg2,arg3)  and so on..

But I'm afraid I can't figure out how to make this work with typeof().


> What these macros would expand to is the following.  We'd generate a
> menu of these for reasonable arities/type combinations and shove them
> into a kernel header.
> 
> #define SYSTEMTAP_PROBE(name) \
>    do { \
>        static void (*__systemtap_probe_##name)(); \
>        if (unlikely(__systemtap_probe_##name)) \
>            (__systemtap_probe_##name) ();  \
>       } while (0)
> #define SYSTEMTAP_PROBE_NS(name,arg1,arg2) \
>    do { \
>        static void (*__systemtap_probe_ns_##name)(int64_t, const char*); \
>        if (unlikely(__systemtap_probe_ns_##name)) \
>            (__systemtap_probe_ns_##name) ((int64_t)(arg1), \
>                                           (const char *)(arg2));  \
>       } while (0)
> 
> As you see, the gist of it is a conditional call through a function
> pointer, where the pointer is in a static variable.  Its name is
> stylized: it encodes the probe name, and its parameter arity/type
> signature.  (It might need some annotations to make sure the compiler
> doesn't elide it, so that it has a convenient alignment, etc.)
> 

The problem with the if() { call } scheme comes when you want to deactivate the
probe. Upon activation, writing the function pointer will be ok for the threads
that pass through this code : they won't do the call until the pointer becomes
non NULL.

When the probe is removed, a thread could clearly end up calling a bad function
pointer by being stalled between the if and the call. You have to take care of
it by making it point to an always valid empty function, but still deactivating
the call with the condition for performance purposes.


> [...] The
> indirect call would be somewhat slower but much simpler than a
> djprobe, and much faster than a kprobe.  It would be great if this
> someone volunteered to microbenchmark this macro family.
> 

But still slower than an inlined function. Inlining has this interesting point :
it doesn't have to build up the function call by putting the arguments on the
stack and does not do the call itself.

If you want to do static tracing, you should really investigate the fastest
solutions available.

In LTTng and LTTV, we use a different approach. We have an "event" code
generator. It takes an event description as input and produces the C logging
function. It can then be easily included as a header with all the versatility of
the C language. Defining a new event becomes as simple as describing the
associated data structure, running genevent, including the header and calling
an inline function to log the event.

If you look for a simple and fast static tracing solution, you might want to
look at LTTng here : http://ltt.polymtl.ca > New features.

I think that both dynamic and static tracers might integrate well together. And
did I say that we have a modular trace viewer (LTTV) ?


Regards,


Mathieu Desnoyers



OpenPGP public key:              http://krystal.dyndns.org:8080/key/compudj.gpg
Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68