Can we do Java probing without invoking stapbm from staprun?

[Tetsuo Handa <penguin-kernel at I-love dot SAKURA dot ne dot jp>]

Hello.

I'm using modified version of /usr/libexec/systemtap/stapbm in order to obtain more
specific values of an object which cannot be obtained by implicit .toString() conversion.
For example, passing $1.getRequestURI() instead of plain $1 when probing
service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
method of javax.servlet.http.HttpServlet class. There should be no problem for SystemTap
side as long as number of arguments matches, for all values are passed to SystemTap side
as string because of PR21020.

Now, I'm trying to go further in order to make

  "allow probing without giving permission to run staprun to the user who runs
   Java application (e.g. tomcat)"

  "automatically start probing as soon as target java process starts (or restarts)
   as with normal userspace probing"

possible by not invoking stapbm from staprun.

Steps for demonstrating such usage is shown below. Can we legally support such usage?



(1) Write a script which hooks nonexistent application name / class name / method name
    and compile the script. Make sure rule name does not get long by specifying module
    name via -m option.

[root@localhost ~]# /usr/bin/stap -p4 -m stap_java -e 'probe java("/").class("class").method("method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String)") { println("Enter " . arg1 . " " . arg2 . " " . arg3 . " " . arg4) }
probe java("/").class("class").method("method_return(java.lang.String)").return { println("Return " . arg1) }'
stap_java.ko

(2) Create a dummy /usr/libexec/systemtap/stapbm in order to confirm rule names
    included in the module compiled at (1).

[root@localhost ~]# cp -p /usr/libexec/systemtap/stapbm /usr/libexec/systemtap/stapbm.orig
[root@localhost ~]# (echo '#!/bin/sh'; echo 'echo "$@" 1>&2'; echo 'exit 0') > /usr/libexec/systemtap/stapbm

(3) Load the module compiled at (1) in order to confirm rule names. In the example
    shown below, the rule names are stap_javaprobe_4113 and stap_javaprobe_4114.

[root@localhost ~]# /usr/bin/staprun stap_java.ko
# /usr/bin/staprun stap_java.ko
install31 / stap_javaprobe_4113 class method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String) 4 entry 0
install31 / stap_javaprobe_4114 class method_return(java.lang.String) 1 exit 0
^Cuninstall31 / stap_javaprobe_4113 class method_enter(java.lang.String, java.lang.String, java.lang.String, java.lang.String) 4 entry 0
uninstall31 / stap_javaprobe_4114 class method_return(java.lang.String) 1 exit 0

(4) Create /usr/libexec/systemtap/stapbm2 based on stapbm. Specify rule names
    confirmed at (3) into echo_bytemanrule().

[root@localhost ~]# cp -p /usr/libexec/systemtap/stapbm.orig /usr/libexec/systemtap/stapbm2
[root@localhost ~]# vi /usr/libexec/systemtap/stapbm2
[root@localhost ~]# cat /usr/libexec/systemtap/stapbm2
#!/bin/bash

# $1 - PID/unique name

exec 1>&2  # redirect byteman/etc. tracing output to stderr, for easier filtering

mode=install
stap="31"
arg_jvmpid=$1

SYSTEMTAP_DIR=${SYSTEMTAP_DIR-$HOME/.systemtap}
BYTEMAN_HOME=${BYTEMAN_HOME-/usr/share/java/byteman}
JAVA_HOME=${JAVA_HOME-/usr/lib/jvm/java}
BYTEMAN_INSTALL_OPTS=${BYTEMAN_INSTALL_OPTS--Dorg.jboss.byteman.transform.all=true}
SYSTEMTAP_VERBOSE=${SYSTEMTAP_VERBOSE-0}

if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
    BYTEMAN_INSTALL_OPTS="$BYTEMAN_INSTALL_OPTS -Dorg.jboss.byteman.verbose"
else
    exec >/dev/null 2>&1
fi

# the byteman and byteman-submit jars should be in ${BYTEMAN_HOME}/lib
BYTEMAN_JAR=${BYTEMAN_HOME}/byteman.jar
if [ ! -r ${BYTEMAN_JAR} ]; then
    echo "Missing $BYTEMAN_JAR"
    exit 1
fi

BYTEMAN_SUBMIT_JAR=${BYTEMAN_HOME}/byteman-submit.jar
if [ ! -r ${BYTEMAN_SUBMIT_JAR} ]; then
    echo "Missing $BYTEMAN_SUBMIT_JAR"
    exit 1
fi

BYTEMAN_INSTALL_JAR=${BYTEMAN_HOME}/byteman-install.jar
if [ ! -r ${BYTEMAN_INSTALL_JAR} ]; then
    echo "Missing $BYTEMAN_INSTALL_JAR"
    exit 1
fi


TOOLS_JAR=${JAVA_HOME}/lib/tools.jar
if [ ! -f ${TOOLS_JAR} ]; then
    echo "Missing $TOOLS_JAR"
    exit 1
fi

# resolve $*prefix fully
prefix=/usr
exec_prefix=/usr
pkglibexecdir=/usr/libexec/systemtap
pkglibexecdir=`eval echo $pkglibexecdir`
pkglibexecdir=`eval echo $pkglibexecdir`

HELPERSDT_JAR=${pkglibexecdir}/HelperSDT.jar
if [ ! -f ${HELPERSDT_JAR} ]; then
    echo "Missing $HELPERSDT_JAR"
    exit 1
fi


# The JVM that invokes byteman will get classpath/etc. settings from
# this shell script to look directly under our $prefix.  However, for
# the target JVM, the HelperSDT* stuff needs to be installed under
# that JVM's paths.

num=`ls -1 ${JAVA_HOME}/jre/lib/ext/HelperSDT.jar ${JAVA_HOME}/jre/lib/*/libHelperSDT_*.so 2>/dev/null | wc -l`
if [ $num -lt 2 ]; then
    echo "Missing HelperSDT JNI class/shared library"
    echo "Install them like this, as root:"
    echo ""
    echo "for so in ${pkglibexecdir}/libHelperSDT_*.so; do"
    echo '  arch=`basename $so | cut -f2 -d_ | cut -f1 -d.`'
    echo "  ln -sf ${pkglibexecdir}/libHelperSDT_"'${arch}'".so ${JAVA_HOME}/jre/lib/"'${arch}'"/"
    echo "done"
    echo "ln -sf ${pkglibexecdir}/HelperSDT.jar ${JAVA_HOME}/jre/lib/ext/"
#   exit 1
fi

flagdir="$SYSTEMTAP_DIR/java"
mkdir -p $flagdir

# Find our target jvm pid.  Due to the possibility of our
# target jvm pid being passed as a string, we need to allow
# for the possiblity that more than one pid may match the
# target jvm pid.  If this is the case, we need to have a
# nested call to stapbm with the actual pid of the jvm pid

if ! [[ $arg_jvmpid =~ ^[0-9]+$ ]]; then
    target_pid=`jps -l | grep $arg_jvmpid | cut -f1 -d" "`
    for target in $target_pid; do
	$0 $target
    done;
    exit 0
else
    target_pid=$arg_jvmpid
fi

# Our target jvm may not have the byteman agent installed yet.  Let's do
# that first.  We use a signal file in $flagdir to show that the
# JVM is ready for further bytemanning without a prior setup step,
# and include in it the designated byteman agent listening-port number.
#
byteman_installed_portfile=$flagdir/`hostname`-${target_pid}-bm

exec 200>>$byteman_installed_portfile # open/create lock file
flock -x 200  # exclusive-lock it

if [ -s $byteman_installed_portfile ]; then
    bmport=`cat $byteman_installed_portfile`

    if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
        echo "Byteman agent reused for java pid $target_pid, port $bmport"
    fi


    # XXX: liveness-check the port; bmsubmit with no argument just lists current rules
    # if fails, delete the _portfile and retry everything
else
    # bmport=9091
    bmport=`expr 9090 + $RANDOM % 10000`
    existing=`netstat -atn | awk '{print $4}' | grep ':'$bmport'$'`
    if [ "x$existing" != "x" ]; then
        echo "Byteman port $bmport already in use, retrying."
        exec "$@"
    fi

# There are two ways to invoke and run byteman operations with the jvm's we're interested
# in, we can alter the startup arguments to include a -javaagent parameter, or use
# byteman and its use of VMAttach libraries, for our case it always makes sense to use
# byteman classes directly and avoid -javaagent

    if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
	echo java -classpath ${BYTEMAN_INSTALL_JAR}:${BYTEMAN_JAR}:${TOOLS_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.install.Install -b -p $bmport $BYTEMAN_INSTALL_OPTS $target_pid
    fi
    java -classpath ${BYTEMAN_INSTALL_JAR}:${BYTEMAN_JAR}:${TOOLS_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.install.Install -b -p $bmport $BYTEMAN_INSTALL_OPTS $target_pid
    if [ $? -ne 0 ]; then
        echo "Byteman agent failed to install for java pid $target_pid, port $bmport"
        exit 1
    fi
    echo $bmport > $byteman_installed_portfile

    if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
        echo "Byteman agent installed for java pid $target_pid, port $bmport"
    fi
    # XXX: Erase file to keep it from sticking around indefinitely,
    # in case process ends, machine reboots, pid gets reused
    # XXX: consider explicit notification to stapbm via process("java").begin/end ?
    # ... or else: liveness-check below
fi
exec 200>&-   # close file & release flock


function echo_bytemanrule()
{
    echo 'RULE stap_javaprobe_4113'
    echo 'CLASS javax.servlet.http.HttpServlet'
    echo 'METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)'
    echo 'HELPER org.systemtap.byteman.helper.HelperSDT'
    echo 'AT ENTRY'
    echo 'IF TRUE'
    echo 'DO METHOD_STAP31_PROBE4("stap_javaprobe_4113", $1, $1.getRequestURI(), $2, "")'
    echo 'ENDRULE'
    echo 'RULE stap_javaprobe_4114'
    echo 'CLASS javax.servlet.http.HttpServlet'
    echo 'METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)'
    echo 'HELPER org.systemtap.byteman.helper.HelperSDT'
    echo 'AT RETURN'
    echo 'IF TRUE'
    echo 'DO METHOD_STAP31_PROBE1("stap_javaprobe_4114", "done")")'
    echo 'ENDRULE'
}


# Generate the byteman rule file on-the-fly
btmfile=$flagdir/`hostname`-$$.btm
echo_bytemanrule > $btmfile
trap 'rm -f $btmfile' 0 1 2 3 4 5 9 15

if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
    echo "Byteman rule file:"
    cat $btmfile
fi

bmcmd=-l

if [ "$SYSTEMTAP_VERBOSE" != "0" ]; then
    echo java -classpath ${BYTEMAN_SUBMIT_JAR}:${BYTEMAN_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.submit.Submit -p $bmport $bmcmd $btmfile
fi

exec java -classpath ${BYTEMAN_SUBMIT_JAR}:${BYTEMAN_JAR}:${HELPERSDT_JAR} org.jboss.byteman.agent.submit.Submit -p $bmport $bmcmd $btmfile

(5) Load the module compiled at (1) in order to start tracing.

[root@localhost ~]# /usr/bin/staprun stap_java.ko

(6) Start target Java process. In the example shown below, tomcat is chosen.

[root@localhost ~]# systemctl start tomcat

(7) Run /usr/libexec/systemtap/stapbm2 in order to enable hooks for tracing
    target Java process (via runuser as needed for switching uid).

[root@localhost ~]# SYSTEMTAP_VERBOSE=1 runuser -u tomcat /usr/libexec/systemtap/stapbm2 org.apache.catalina.startup.Bootstrap
java -classpath /usr/share/java/byteman/byteman-install.jar:/usr/share/java/byteman/byteman.jar:/usr/lib/jvm/java/lib/tools.jar:/usr/libexec/systemtap/HelperSDT.jar org.jboss.byteman.agent.install.Install -b -p 16175 -Dorg.jboss.byteman.transform.all=true -Dorg.jboss.byteman.verbose 12131
byteman jar is /usr/share/java/byteman/byteman.jar
Byteman agent installed for java pid 12131, port 16175
Byteman rule file:
RULE stap_javaprobe_4113
CLASS javax.servlet.http.HttpServlet
METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
HELPER org.systemtap.byteman.helper.HelperSDT
AT ENTRY
IF TRUE
DO METHOD_STAP31_PROBE4("stap_javaprobe_4113", $1, $1.getRequestURI(), $2, "")
ENDRULE
RULE stap_javaprobe_4114
CLASS javax.servlet.http.HttpServlet
METHOD service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
HELPER org.systemtap.byteman.helper.HelperSDT
AT RETURN
IF TRUE
DO METHOD_STAP31_PROBE1("stap_javaprobe_4114", "done")")
ENDRULE
java -classpath /usr/share/java/byteman/byteman-submit.jar:/usr/share/java/byteman/byteman.jar:/usr/libexec/systemtap/HelperSDT.jar org.jboss.byteman.agent.submit.Submit -p 16175 -l /usr/share/tomcat/.systemtap/java/localhost.localdomain-12215.btm
install rule stap_javaprobe_4113
install rule stap_javaprobe_4114

(8) Confirm that trace output is sent to staprun side started at (5) when hooks
    with application name specified at (7) and class name / method name specified
    at (4) are called.

Enter org.apache.catalina.connector.RequestFacade@44009cf2 /tomcat.css org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /tomcat.png org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /asf-logo-wide.svg org.apache.catalina.connector.ResponseFacade@1f3c368
Enter org.apache.catalina.connector.RequestFacade@44009cf2 /bg-nav.png org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Return done
Enter org.apache.catalina.connector.RequestFacade@7b9aeef0 /bg-button.png org.apache.catalina.connector.ResponseFacade@78bd0df5
Enter org.apache.catalina.connector.RequestFacade@43df8dc0 /bg-upper.png org.apache.catalina.connector.ResponseFacade@9d35b10
Return done
Return done
Enter org.apache.catalina.connector.RequestFacade@12ff7c7a /favicon.ico org.apache.catalina.connector.ResponseFacade@43453754
Return done
Enter org.apache.catalina.connector.RequestFacade@75338e4c /bg-middle.png org.apache.catalina.connector.ResponseFacade@bc35bb1
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/ org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/ org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@12ff7c7a /examples/servlets/images/code.gif org.apache.catalina.connector.ResponseFacade@43453754
Return done
Enter org.apache.catalina.connector.RequestFacade@44009cf2 /examples/servlets/images/return.gif org.apache.catalina.connector.ResponseFacade@43ff6bf
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/images/execute.gif org.apache.catalina.connector.ResponseFacade@1f3c368
Return done
Enter org.apache.catalina.connector.RequestFacade@25b2dd9e /examples/servlets/servlet/HelloWorldExample org.apache.catalina.connector.ResponseFacade@1f3c368
Return done