Re: documentation for user-space usage?

[Grant Edwards <grant dot b dot edwards at gmail dot com>]

On 2010-10-31, Frank Ch. Eigler <fche@redhat.com> wrote:

> grant.b.edwards wrote:
>
>> I've figured out how to do kernel-tracing on an embedded system by
>> cross-compiling .stp files and then running them on a target that has
>> only staprun installed.
>
> Great.  (Were you doing this by manual copying, or via the stap-server
> code?)

I copy the compiled script to the target via scp/tftp/etc. and then
run it with the staprun command.

>> [...]
>> I want to put logging statements in user application source code that
>> log various events along with maybe one or two parameter values along
>> with an event.  I need to log up to a few thousand events/second for
>> short periods of time into a circular buffer and then transfer the raw
>> log data to a different machine for display/analysis.  I need to start
>> stop tracing and enable/disable various subsets of those trace points
>> at runtime (hopefully without having to restart the processes doing
>> the logging).
>
> All those look within the realm of systemtap (and probably some other
> tools).

One of the other options I'm looking at is LTTng, but the user-space
stuff has apparently never been tested on ARM, and if it does work the
user space event tracing is going to involve a context switch to fetch
a timestamp and even then you won't be able to correlate kernel-space
and user-space events.  LTTng also appears to be pretty heavyweight
and complicated.  For each thread/process that you want to instrument,
you have to have a companion thread that hands the thread's log data
off to a kernel module which then hands it off to daemon, which then
writes it to disk.

>> About the only documentation I can find on systemtap user-space stuff
>> is http://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToApps,
>
> The beginner's guide is going to get more content about this soon.
> In the mean time, stapprobes(3stap) includes documentation.  See
> the
>      process("/path/to/binary").{statement,function,mark}(...)
> probes.

Assuming utrace/uprobes gets ported to ARM, does such a trace events
involve a context switch to kernel-mode and then back to user-mode?

>> I was under the impression that uprobes support was required, but
>> that page says only mentions utrace as a requirement.
>
> uprobes (for several platforms) is currently provided as a part of
> the systemtap distribution.  It prereqs utrace in the kernel.
> Depending on your architecture and kernel distribution of interest,
> this could be all done, or else a moderate amount of porting work.

I'm using ARM9.  Utrace support patches have apparently been rejected
by the kernel maintainers, so I'd hae to maintain my own fork of the
kernel as well as port uprobes.  I think.

>> All the examples I can find of user-space probes seem to require that
>> stap be installed on the machine where the probed user-processes are
>> being run.  [...]
>
> User-space probing and kernel-space probing are basically identical
> from the point of view of the stap user.

Is the timing impace of a user-space probe no different than that of a
kernel-space probe?

> You can cross-compile probes the same way.
>
>> Am I on the right track looking at systemtap for user application
>> event logging, or not?
>
> The theoretical fit is pretty good.  If in practice you are blocked
> by some missing unported layer, you could decide between helping
> port, prototyping on x86 while someone else ports, and/or
> investigating other logging-related tools/libraries.

I wouldn't mind working on porting uprobes if I was confident that it
would be accepted upstream.  Since utrace support was apparently not
accepted, I'm not too optimistic.

-- 
Grant Edwards               grant.b.edwards        Yow! Does someone from
                                  at               PEORIA have a SHORTER
                              gmail.com            ATTENTION span than me?