Systemtap vs Dtrace web page corrections.

["James Dickens" <jamesd dot wi at gmail dot com>]

This is in response to this web page posted

http://sources.redhat.com/systemtap/wiki/SystemtapDtraceComparison

I have went line by line and commented on its content and in accuracies.

Going line by line...

Licenses: are correct.

Processor support:
Not sure if Apple, or  the FreeBSD guys ported to PPC or not, but if
Apple doesn't support, the Polaris project will support DTrace on its
PPC OpenSolaris port.
Of course it should be pointed out that Systemtap only supports Linux,
where Dtrace now supports Solaris and OSX, with the FreeBSD  DTrace
port rapidly proceeding, in fact last I checked they are furher along
in the process that Systemtap is.

Kernel Lock in:
DTrace kernel lock step is highly limited when compared to Systemtap.
Remember that Linux doesn't believe in API/ABI stability. I just ran a
complex DTrace script I wrote over 2 years ago on a modern Solaris
express release and it worked just as it always has,

Systemtap isn't even stable enough to run a complex script written 6
months ago, even if you were on the same kernel, of course you have to
update your elfutils quite often as well, I don't really see anything
that is stable about Systemtap. So on this line, it would be that
Systemtap has the greater amount of lock in, not just in the kernel
but Systemtap it self and possibly even the tools used to compile the
script.

Core Developers:
The majority core developers and active contributors on the Systemtap
mailing list are IBM and RedHat employees.

DTrace is currently developed by a number of people outside of sun,
including the people that have ported DTrace to OSX, as well as the
team that is working on porting it to FreeBSD. There are also many
users of DTrace that submit bug reports and static probes for
scripting languages and applications.

Development began:
Are you accounting for kprobes that was the base of Systemtap
developed for a while before Systemtap started?
DTrace development began in October 2001, and first released in
September 2003 along with submission of the USENIX paper.

Ongoing evolution:
Yes Systemtap is development is rapid, but at the same time they claim
to have no kernel lock in? Exactly which is it, we have Linux that has
no guaranteed API/ABI (rapidly changing) including the kprobes kernel
module that is still under active development, and a project that is
rapidly changing, so how can it have little kernel lock in? Of course
Systemtap needs to have rapid evolution to try and match where DTrace
is today.

DTrace on the other hand, has met most of its goals and is now adding
features and fixing bugs, where the active development is adding
static probes to languages and apps making it easier for the end user
and developers. DTrace currently has accomplished 99% of its goals it
stated at the beginning of the project
(http://www.usenix.org/event/usenix04/tech/general/full_papers/cantrill/cantrill_html/)

Target Audience:
Systemtap has seemed to miss its target audience, its current audience
is Kernel developers; I know of no users that use Systemtap on a daily
basis, the same statement holds  true for Sysadmins as well. There are
hardly any pre-made Systemtap scripts, at the moment so your only
users are kernel Coders. I've not heard of any userland developers
using Systemtap to solve problems since userland probes are not
included. Nor is support for any other application or scripting
language probing.

DTrace is targeted at Developers including ones that specialize in
kernel and userland, System administrators, and end users. DTrace is
so stable that they include scripts that are providing performance
data as part of the operating system that are used daily in production
systems with out fear. Brendan Gregg's DTrace tool kit provides ready
made script so that even newbies that can't program a line can benefit
from DTrace. DTrace is also working to integrate static probes in
scripting languages including Java, Ruby, Php, Perl, Python, Postgres,
Apache, Xserver these and one line DTrace script allows end users to
figure out performance bottlenecks in applications even though they
can't code in either DTrace or the target language.

Target Usage:
DTrace is targeted at debugging, well if it wasn't it sure does a job
great at it anyway, please see
http://uadmin.blogspot.com/2006/05/what-is-dtrace.html for examples of
bugs being solved by DTrace, these bugs are not just limited to
Solaris applications, DTrace has been used to solve new and long
existing bugs in userland applications including, NTP, Gnome, Java
Applets, Mozilla, Star Office/Open Office. This is not a complete list
just the ones I dug up on the internet a while ago.
Systemtap seems limited to solving Linux Kernel problems since it has
no userland probes, and no non-kernel developers or system
administrators using it on a daily basis.

Style:
Since when did C language turn into a scripting language? Your scripts
are basically C code modified to make it compatible with aC compiler.
If one turns on guru mode, you are writing 100% C code, no way to
consider it anything but C.

Control structures:
Systemtap has full control structures as stated, but if a bug happens
in the Systemtap script it can cause the box to crash.
DTrace: doesn't have functions or loops per se, but you can work
around this with a little thought.  In return even if a DTrace user
makes a horrendous mistake, it doesn't take down the box. Seems like a
fair trade, doesn't it? It does to me, you should ask your target
users, are any of them perfect? In that they never make a mistake? Are
they willing to crash there production box because of a mistake in
their systemtap script?

Variable Typing:
Systemtap's use of implicit type control seems more of a limitation
that a feature,
Because by implicitly setting the type of the variable you are now
locked into that type, if the code you are probing changes a variable
type your script no longer works. Thus eliminates the possibility of
providing ready made scripts long term. This is made even worse since
Linux doesn't have a stable API or ABI, so the script developed today
will have to be recompiled and possibly ported with the next kernel
release or the next release of a program.
When Systemtap gets userland probes its inferred variable typing will
become even more of a hindrance, lots of programs don't ship with
debugging code embedded, so if you don't have the source code to
recompile with debugging information Systemtap is useless, with DTrace
it will try and make guesses at the data structure and include files
and allow the user to type cast variables as needed, Systemtap does
not have the native ability to process include files or handling data
of unknown types.

Complex Reports:
If Systemtap's report generating ability is so great why is there work
done on a dashboard that is designed to make the reports look better?
What limitations are you seeing in DTrace's report generating capability?

Thread-local variables:
Systemtap: seems to have added these as an after thought, as justified
by the need for disclaimer.
DTrace this is a built in feature, not an after thought.

Speculative tracing:
Systemtap: you can't just wish this requirement a way, you can't judge
whether or not you need a piece of data, in a complex system when the
first event occurs, you have to store it  until possibly  many other
events have occurred, and so unless your script can predict the
future, it really should provide a way to abandon information it
collected that it really didn't care about. Your users may not feel
confident enough with the framework to write complex scripts, that
need this functionality, where this is a problem.

Binary Tracing:
This seems vague at best, and an unimportant implementation detail
that the end user really won't care about.

Probe execution:
For Systemtap its compiled C code, especially in guru mode.

Number of probe points:
System tap, has too much bloat to have a truly unlimited number of
probes, the overhead of each probe, last I checked was 64 bytes of
code per script code segment, attached to a probe at minimum, plus
additional storage requirements for each probe, so it does have a
finite number of probes, 1 million active probes requires 64MB of
kernel space allocated, since each Systemtap module is independent of
all others, you get multiple copies of kernel code that is shared in
DTrace.
What is the highest number of Systemtap probes ever activated in an
active script with out the machine falling over?  In my tests of
DTrace it is over 500,000.  The reason for stopping at 500,000 wasn't
because I hit any limit in the system, just decided any more was
pointless. I could easily run multiple copies of the same script with
out a problem. By the way the limit of 50,000 pre-defined probe points
in DTrace are just for the kernel, you can probe any line of userland
code, with out limit. Really for all but hard core kernel coders it is
an unlimited number of probes because 99.99% of your users will not
need to probe a specific line in the kernel.

Of course a good question is, what kind of kernel coder can't debug a
problem when he knows what functions are being called, and by what
function, how often and how much time its taking, and what functions
it calls and complete userland and kernel stack tracing. Seems to be a
pretty silly to do all that extra work and risk system stability in
the name of probing every line in the kernel.

Probe arbitrary points in code:
DTrace can probe entry and exit points of any function whether it is
userland or kernel space, you can also define static probe points in
both userland and kernel code, Arbitrary probe points in kernel code
really seems to be of very limited importance by 99.9999% of coders as
explained in the explanation above. DTrace can probe arbitrary points
in userspace applications should it be required.
Can Systemtap still probe arbitrary points in code if you don't have a
binary with debugging information intact?

Dynamic loaded kernel objects
Yes DTrace can probe dynamically loaded kernel module, as well as
userland libs that are loaded dynamically.

Concurrent probes on multiprocessors:
DTrace can probe multiple processors, and multiple tasks and multiple
threads regardless if they are userland or kernel space. You can even
run multiple copies of the same script with out problems, last time I
tried this simple test on Systemtap, it failed.

Extract arbitrary data at probe point:
DTrace can read any location in memory be it in kernel space or
userland, during probe execution. DTrace also handles any traps that
occur as the result of the attempt of reading the data on its entire
platforms, last I heard this feat has not been accomplished in all of
SystemTap's platforms. DTrace also understands the C struct construct
so it can also read data stored in structs and use pointers stored in
structs to access other data even if the program isn't compiled with
debugging information. Systemtap requires debugging information to
access data in structures even if the user knows the layout of the
data structures.

End user extendible libraries:
Systemtap is at the same state as DTrace script is a compiled program
requiring a compiler on the target system; there is no way for a user
to extend the probe library with precompiled scripts that are
distributed without a compiler installed.

Hardware performance counters probing:
DTrace is working on adding access to performance counters that are
available in the processors.

Safety:
The safety category is correct mostly on the DTrace side.

The Systemtap side, it talks a good game, but the proof is in the
pudding. Even given Systemtap's limited use in by the general public,
when was the last week, that no bug report was filed against Systemtap
that involved the system falling over. In the last 2 years, DTrace has
perhaps had a handful of such reports.
Has anyone used Systemtap on a daily basis to solve problems on
production systems without fear of it falling over?

James Dickens
uadmin.blogspot.com