RE: tapset taxonomy

["Chen, Brad" <brad dot chen at intel dot com>]

>Naturally, such power would be left to "guru-mode" users and to
>privileged script "tapsets" located in a library.
This notion of priviledged script "tapsets" in a tested and 
trusted library is a critical one for me. It will be a common
occurence that people try to run buggy, untested scripts, so
the script authoring environment needs to be constrained. In
contrast it should be extremely rare that people encounter a 
buggy tapset, as rare as bugs in the Systemtap runtime, 
provided that tapset is part of the Systemtap release. It should
also be relatively rare that systemtap users write their own 
tapset, provided the built-in tapsets approach the completeness 
of the DTrace providers. The testing, reliability, and trust 
assumptions about tapsets implementations are entirely different 
from those for scripts, and this is a relevant design 
consideration for the corresponding authoring environments.

>...  Those needing to interact
>with future peculiar chunks of kernel (e.g., Brad's list of future
>instrumentables) likely need to emit strange new C code to talk to
>them, so need language and/or translator extensibility.
>
>In order to let us decide how many of these extensibility points we
>need to support, we need ... drumroll ... more detailed examples.  My
>intuition says that the basic library and translator points I marked
>above with [***] are sufficient for the forseeable future.
>notion of priviledged script tapsets located in a library
I think the list of mine you refer to above is this one:
- hardware performance counters
- power management hardware
- virtualization technologies
- strange new hardware to support multicore architectures
Unfortunately, this is not the future; the first three are happening
now, and the fourth will be with us within a year or so. Dealing with
some of these can require special libraries, linking conventions or 
compilation options. It's easy to imagine the compiler support for 
such options might be somewhat tempermental and inflexible at first.
I don't think we will understand all the implications before we commit
to a plan for tapset authoring.

Because the safety considerations for tapsets are more like those of
the runtime than of scripts, and because we are going to be facing a 
lot of relevant hardware features we don't know how to deal with yet, 
I'm concerned about overly-constraining the tapset authoring 
environment. It's great if tapsets are commonly implemented entirely 
in script, but it shouldn't be a requirement. The library, language, 
and translator extensibility options you describe below sound like 
useful abstractions, but I don't see the justification for prohibition 
of C.

Brad

-----Original Message-----
From: systemtap-owner@sources.redhat.com
[mailto:systemtap-owner@sources.redhat.com] On Behalf Of Frank Ch.
Eigler
Sent: Thursday, April 07, 2005 2:10 PM
To: systemtap@sources.redhat.com
Subject: tapset taxonomy

Hi -


Here is my attempt to put into words an insight from today's call.
The core issue is extensibility: how to extend the reach of systemtap
script into the target software.  This reach can be some combination
of data access and function calling.

Extensibility can occur within the library, within the language, or
within the translator.  Their nature and availability has implications
for safety, utility, and tapset authorship.

Library extensibility refers to the availability of a library of
script fragments, which build upon existing language/translator
mechanisms to build abstractions for end-user scripts.  These
effectively abbreviate the user script, and provide no extra
expressivity. [***]

Language extensibility may be an syntactic embedding mechanism within
the scripting language, like explicit C code in yacc grammars, or like
in dpcc's expression strings.  If the translator passes such
constructs through, then script authors might get approximately the
same unconstrained expressive power as someone writing C code.
Naturally, such power would be left to "guru-mode" users and to
privileged script "tapsets" located in a library.

Translator extensibility may occur by some mixture of translator
changes, such as recognizing additional probe-point specifications,
automagically providing synthetic variable/function declarations to
user scripts.  Since these all involve changes to the translator, they
naturally enable the extension author to emit arbitrary lower-level C
code to enable the higher-level additions.  In this case, the script
user doesn't have any extra syntax, and is constrained to touch only
whatever data the translator modifications deemed fit.  The emitted
lower-level C code could make use of just the base kernel.  Or it
could refer to additional kernel modules, and maybe even extra local
libraries, as the extension author sees fit.  These additional kernel
modules or libraries don't need a-priori fixed APIs, since the
modified translator can talk to them on their own terms. [***]

One variant of this may occur by a plug-in interface that allows
someone to load new modules into the translator.  If pursued, this
mode could allow extension along the above lines, but without
requiring translator source code changes.

Another variant of this constitutes a middle road between language and
translator extensibility: debug-info processing.  The translator can
grok the dwarf debug-info of modules relevant to given probe-points,
and can provide constructs (probably function declarations) to the
scripts to traverse them.  This should give around the same
expressivity as the gdb/dpcc style has, but with more generality,
language-neutrality, control and (sadly) probably wordy
verbosity. [***]


So, where does all this leave "tapset" authors?  Each will decide
which of the available points of extensibility they must exploit in
order to expediently expose their excellent expertise.  Some may
settle for vanilla script libraries only.  Those needing to interact
with future peculiar chunks of kernel (e.g., Brad's list of future
instrumentables) likely need to emit strange new C code to talk to
them, so need language and/or translator extensibility.

In order to let us decide how many of these extensibility points we
need to support, we need ... drumroll ... more detailed examples.  My
intuition says that the basic library and translator points I marked
above with [***] are sufficient for the forseeable future.


- FChE