Probing user-space markers in RPM installed binaries

[William Cohen <wcohen at redhat dot com>]

I did an experiment this evening to see if can get user-space probing
to work with executables in RPMs. Summary: I was able to get things to
work on Fedora 10 machine with the 2.6.27.9-159.fc10.x86_64 kernel and
postgresql-8.3.5-1.fc10 on an x86_64 machine with only minor tweaks to
the postgresql spec file.

Downloaded the latest git version of systemtap that included Stan's
checkins with sdt.h and dtrace. Did the usual configure in a build
directory and then created a tarball with:

make dist-gzip

Copied the systemtap-0.8.tar.gz into the rpm SOURCES directory and
copied the systemtap.spec into the rpm SPEC directory.  Built the rpms with:

rpmbuild -ba systemtap.spec

With the recent Stan's recent checkin the rpmbuild will generate
systemtap-sdt-devel rpm. Install systemtap-sdt-devel and the other
systemtap rpms on the machine. The systemtap-sdt-devel rpm will be
needed when building the other RPMs with userspace markers.

The changes for the postgresql spec file were very modest:

--- postgresql.spec	2008-11-02 13:23:29.000000000 -0500
+++ postgresql.spec.new	2009-01-13 23:15:07.000000000 -0500
@@ -75,6 +75,7 @@
 %{!?pam:%define pam 1}
 %{!?pgfts:%define pgfts 1}
 %{!?runselftest:%define runselftest 1}
+%{!?sdt:%define sdt 1}

 # Python major version.
 %{!?python_sitearch: %define python_sitearch %(%{__python} -c "from 
distutils.sysconfig import get_python_lib; print get_python_lib(1)")}
@@ -157,6 +158,10 @@
 BuildRequires: pam-devel
 %endif

+%if %sdt
+BuildRequires: systemtap-sdt-devel
+%endif
+
 # main package requires -libs subpackage
 Requires: postgresql-libs = %{version}-%{release}

@@ -439,6 +444,9 @@
 %if %pgfts
 	--enable-thread-safety \
 %endif
+%if %sdt
+	--enable-dtrace \
+%endif
 	--with-system-tzdata=/usr/share/zoneinfo \
 	--sysconfdir=/etc/sysconfig/pgsql \
 	--datadir=/usr/share/pgsql \


The postgresql RPM are built now can be built with userspace markers
with the following line:

rpmbuild -ba postgresql.spec --define 'sdt 1'

Install the resulting binary rpms. Access to the arguments of the
markers REQUIRES the postgressql-debuginfo RPM to be installed.

The example from the wiki page
(http://sourceware.org/systemtap/wiki/UsingStaticUserMarkers) needs to
be tweaked because the binaries is installed in /usr/bin/postgresql:


probe process("/usr/bin/postgres").mark("transaction__start")
{
  printf("%s %#x\n", "transaction__start", $arg1);
}
probe process("/usr/bin/postgres").mark("transaction__commit")
{
  printf("%s %#x\n", "transaction__commit", $arg1);
}
probe process("/usr/bin/postgres").mark("transaction__abort")
{
  printf("%s %#x\n", "transaction__abort", $arg1);
}
probe process("/usr/bin/postgres").mark("lock__startwait")
{
  printf("%s %#x %#x\n", "lock__startwait", $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lock__endwait")
{
  printf("%s %#x %#x\n", "lock__endwait", $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lwlock__endwait")
{
  printf("%s %#x %#x\n", "lwlock__endwait", $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lwlock__acquire")
{
  printf("%s %#x %#x\n", "lwlock__acquire", $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lwlock__condacquire__fail")
{
  printf("%s %#x %#x\n", "lwlock__condacquire__fail",
         $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lwlock__condacquire")
{
  printf("%s %#x %#x\n", "lwlock__condacquire", $arg1, $arg2);
}
probe process("/usr/bin/postgres").mark("lwlock__release")
{
  printf("%s %#x\n", "lwlock__release", $arg1);
}


Inialize the postgresql database:

/usr/bin/initdb -D /tmp/data


Then run the little experiment with:

PGDATA=/tmp/data stap -c '/usr/bin/postgres' postgres.stp

Get the following output (hit control-c to stop):

lwlock__release 0x0
lwlock__acquire 0x20 0x0
lwlock__release 0x20
lwlock__acquire 0x79 0x0
lwlock__release 0x79
lwlock__acquire 0x7a 0x1
lwlock__release 0x7a
lwlock__acquire 0x7a 0x1
lwlock__release 0x7a
lwlock__acquire 0x7a 0x1
lwlock__release 0x7a
lwlock__acquire 0x7a 0x1
lwlock__release 0x7a
lwlock__acquire 0x3 0x0
lwlock__release 0x3
lwlock__acquire 0x23 0x1

-Will