Re: Phase 2 takes too much time to complete

[fche at redhat dot com (Frank Ch. Eigler)]

Roberto Natella <rnatella@gmail.com> writes:

> [...]  On my system (RHEL4 with 2.6.25 kernel; dual Intel Xeon(TM)
> CPU 2.80GHz with HT; 5 GB RAM), phase 2 takes about half an hour to
> complete. [...]

That's wacky!


> SystemTap 0.8 and in most recent snapshots.  Using strace, I found
> that stap spends most of time for generating the header
> "stap-symbols.h" in function dump_unwindsyms() (in translate.cxx).

There are several sources of delay in there.

First, this code re-traverses the kernel module tree looking for
.ko's, which is basically a "find /lib/modules/.../" that can chew
through a lot of inodes.  Instead, this information should be
extracted & saved the first time we go through the modules (in pass 2
rather than pass 3).

Second, this code uses "endl" a lot, which can result in premature
buffer flush syscalls.  If your strace is showing a lot of small
individual writes when producing this data, then it's worth trying to
replace '<< endl' with '<< "\n"' throughout this function.

Third ... well, let's check in with an strace or even a oprofile/gprof
data first.


> [...] Because the symbol list remains the same if the kernel is
> unchanged, I think that stap-symbols.h should be cached in order to
> speed up phase 2. You can find a very simple patch I made on
> SystemTap 0.8 which caches stap-symbols.h [...]

That's not a bad idea at all, and thanks for working out a patch to do
it, but I think we should be able to solve it somewhat differently.
(For one thing, the list of modules included in there can include
userspace binaries and different subsets of the kernel modules, so the
cache key can't be just a function of the kernel image.)


- FChE