Re: [RFC] SystemTap future direction

[Masami Hiramatsu <masami dot hiramatsu dot pt at hitachi dot com>]

Frank Ch. Eigler wrote:
> masami.hiramatsu.pt wrote:
> 
>> As you may know (of course I Cc'd discussion on LKML), Ingo and
>> Christoph said that (at least) uprobes (but also kprobes) should not
>> support out-of-tree module.
> 
> This is unfortunate, but is in a way useful evidence about what sorts
> of effects we can expect from further outreach efforts.  You and
> Srikar have spent at least a year working on mini-stap functionality
> for the benefit of the perf side of the house, and yet the amount of
> goodwill offered in return is ... where?

Well..., there may be still less use-cases they know.

>> This means that if we succeed to merge uprobes into kernel,
>> SystemTap can't use uprobes itself.
> 
> Well, there are always various social and technical measures to work
> around gratuitious obstacles.
> 
>> Even worse, if someone tries to remove kprobes' module support, that
>> could shake the foundation of SystemTap.
> 
> It is hard to imagine someone deliberately hurting linux users that way.
> 

I hope so. and now I know we already have many users, use-cases.
I think the next step is let them know the usefulness of systemtap,
with actual use-case examples.

> 
>> At least, to add support kmodules to uprobes, I think we have two
>> options, one is pushing systemtap itself and useful scripts into
>> kernel tree, or the other is finding very useful use-case of *probes
>> which requires out-of-tree module.  [...]
> 
> Or #3, coming up with one more substantial in-tree uprobes example
> than the one hch instructed srikar to drop.

OK.

>> [...]
>> I'd like to suggest some directions here;
>>
>> - Merge runtime and module-source generator into linux kernel.
>>   This will requires rewriting whole of systemtap code from C++ to
>>   C or other LL (perl or python)
> 
> More concretely, to rewrite and LKML-code-standarize the lot, but
> retain current architecture?  Do you sense that there's any interest
> in this sort of solution by Linus?

Not sure. If the on-line scripting (which requires module-generator,
or in-kernel interpreter) is so useful as you think, you can convince
him to support it in-kernel.

Again, we already have several users who are using systemtap (with
Dtrace tracepoints) agressively. I think that means there are real
demands of application tracing. (I also think we still need to discuss
what is the best implemantation for that.)

>> - Port SystemTap on the perf/ftrace and extend perf/ftrace to support
>>   extend handlers which provided by modules.
> 
> More concretely, to make a version of systemtap that instead of
> generating stand-alone kernel modules that operate independently of
> perf/etc., that they be bound to perf event sources & infrastructure?

Right.

> But retain the power of our system by still executing arbitrary
> generated code from those callbacks?  Do you sense that there's any
> interest in this sort of solution by the perf people?

I think so, if we can indicate the flexibility of on-line scripting
and its power, we may convince them.

> Now if we're talking about a module-encased bytecode interpreter / JIT
> rich enough to encompass our runtime/language features, I have some
> interest in this sort of solution, whether coupled or decoupled from
> perf.  But this is a large amount of effort.  But we're tempted.

Yeah, me too :)
Actually, Ingo had once suggested to build an interpreter in kernel.

>> - Port SystemTap on the perf/ftrace but drop embedded-C support.
>>   This will enhance perf/ftrace to support enough flexible data
>>   filter/modifier (including fault injection feature). In this case,
>>   SystemTap scripts will handle the data in user-space (not on-line).
> 
> I get the sense the perf people believe they are on this course
> already, without needing any help.

Yeah, I just would like to help systemtap users to run their
stap scripts even if perf people choose this way.


>> - Or, just do nothing and wait for kernel maintainers choking
>>   our necks...
> 
> I don't think the situation is in fact deteriorating.  We're shipping
> decent releases, growing our user base, within and without the kernel
> developer community, and still have plenty of major feature areas to
> work on.  We have not seen regressive LKML obstructions, though
> admittedly that is a low standard when it comes to serving the
> community.

Maybe I'm a paranoid. However, I can't see the things getting better
too, at least in the kernel mailing list. perf and ftrace are
becoming to de-facto standard tracing tools among the linux kernel
community. Indeed systemtap has growing user base and major features.
And then, why NOT appeal that? I can't see any systemtap developers
in the collaboration summit this year, and on LinuxCon attendee list.
Recently I feel that we are losing presence among the linux kernel
community, even though systemtap can run ONLY on the linux.

Anyway, I plan to hold a tracing panel discussion with systemtap
users and perf/ftrace developers in LinuxCon Japan and will try
to clarify what features users need. I hope that helps things
going forward.

Thank you,

-- 
Masami HIRAMATSU
2nd Research Dept.
Hitachi, Ltd., Systems Development Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com