Re: Systemtap userspace performance

[fche at redhat dot com (Frank Ch. Eigler)]

ju wrote:

> [...]
> I'm evaluating the performance of systemtap userspace instrumentation
> (with utrace) and I'd like to make sure the procedure I use is the right
> one to get the best results.

OK.

> I'm particularly interested in the scalability of the solution to
> multi-core and multi-processor systems.

OK.  I recall there is at least one known scalability-related obstacle
in the current implementation of uprobes/utrace.
http://sourceware.org/bugzilla/show_bug.cgi?id=5660

Note that the rest of your message does not deal with scalability per
se (how the performance varies with number of processors), just
single-processor overhead.


> My test program creates threads which have for only purpose to call a
> function which calls 10 million times an other function [...]
> Right now I'm only working with 1 thread, my baseline (running the
> program without instrumentation) gives me 0:00.27 of elapsed time (0.27
> second).

OK.

> probe process("./benchmark").function("single_trace") [...]
> 0:51.82 elapsed time.
> [...]
> probe process("./benchmark").mark("single_trace") [...]
> I get 0:27.60 elapsed time 

These numbers are on the order of 2-5 microseconds per hit.  They are
not grossly out of whack with what we've seen before.  What are you
comparing this amount to?

> [...]  I mesured the gettimeofday_ns() is responsible for 8.4
> seconds for each test.

We should be able to improve on that.  Try get_cycles() instead.

Also be aware that the formatted printing is also of considerable load
(and is distinct from whether you ultimately store the data on disk or
not).  You might try measuring just printf()'s contribution.  We have
plans to speed that up.
http://sourceware.org/bugzilla/show_bug.cgi?id=3217

- FChE