This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
How to use the new upcoming stapvirt binary for VM probing
- From: Jonathan Lebon <jlebon at redhat dot com>
- To: systemtap at sourceware dot org
- Date: Thu, 26 Sep 2013 17:13:35 -0400 (EDT)
- Subject: How to use the new upcoming stapvirt binary for VM probing
- Authentication-results: sourceware.org; auth=none
Hello everyone,
We've been working on a new feature of SystemTap which will
allow users to execute scripts within virtual machines managed
by libvirt. Put briefly, this would allow you to do this from
the host:
[jlebon ~]$ stap script.stp --remote=libvirt://MyVM
This does not use ssh, but a virtio-serial channel, which means
that the VM can have its networking disabled. The virtio-serial
channel is also slightly faster than ssh in most regards for
large amounts of data.
I recently put up a mini-session on the related PR (13078)
demonstrating how it will be used. Here it is inlined for your
viewing pleasure. :) Feedback is always welcome.
Cheers,
Jonathan
--------------------------------------------------------------
# This console session explains how one would use stapvirt and the new
# libvirt:// scheme to execute SystemTap scripts inside virtual machines managed
# by libvirt.
# First I'll add my custom stap install to the path
[jlebon ~]$ PATH=codebase/systemtap/install/bin:$PATH
[jlebon ~]$
# Let's run stapvirt
[jlebon ~]$ stapvirt
stapvirt v2.4
Usage: stapvirt [-c URI] [-d PATH] [-v] COMMAND ARGUMENTS
-c Specify the libvirt driver URI to which to connect [default: NULL]
-d Specify the directory in which sockets should be placed [default:
/var/lib/libvirt/qemu]
-v Increase verbosity
Try the 'help' command for more information.
[jlebon ~]$
# OK, let's try the 'help' command to see what it can do
[jlebon ~]$ stapvirt help
stapvirt v2.4
Usage: stapvirt [-c URI] [-d PATH] [-v] COMMAND ARGUMENTS
-c Specify the libvirt driver URI to which to connect [default: NULL]
-d Specify the directory in which sockets should be placed [default:
/var/lib/libvirt/qemu]
-v Increase verbosity
Available commands are:
help
Display this message.
list
List available domains.
port-add <domain>
Add a permanent SystemTap port to the domain's definition. If the
domain is currently running, it must be restarted before changes take
effect.
port-list <domain>
List the UNIX socket paths of the permanent SystemTap ports in the
domain's definition.
port-remove <domain>
Remove a permanent SystemTap port from the domain's definition. If
the domain is currently running, it must be restarted before changes
take effect.
query <domain>
Display the following information about the domain: its name, its
UUID, its state, the number of permanent SystemTap ports installed,
and whether hotplugging is supported.
Domains can be specified using their name, UUID, or ID.
[jlebon ~]$
# Sweet! Let's see what VMs (aka domains) we have on this machine
[jlebon ~]$ stapvirt list
Available domains on URI 'qemu:///session':
UUID Name State Type
[jlebon ~]$
# Right, the default libvirt URI is NULL, which means that on e.g. f19, it will
# default to qemu:///session. This is the same as virsh:
[jlebon ~]$ virsh list --all
Id Name State
----------------------------------------------------
[jlebon ~]$
# Most likely, your domains reside on qemu:///system:
[jlebon ~]$ virsh -c 'qemu:///system' list --all
# (You get authenticated)
Id Name State
----------------------------------------------------
5 TestVM running
[jlebon ~]$
# Let's try the same thing with stapvirt
[jlebon ~]$ stapvirt -c 'qemu:///system' list
Available domains on URI 'qemu:///system':
UUID Name State Type
905951c0-fa4f-409b-079c-c91ddda27028 TestVM running (ID 2) persistent
[jlebon ~]$
# Rather than repeating the -c command everytime, you can either set the
# LIBVIRT_DEFAULT_URI env var to 'qemu:///system', or run everything as root.
# Let's do the latter so that we also don't have to authenticate:
[jlebon ~]$ su
Password:
[root jlebon]#
# OK, now let's try the query command to see more details about TestVM
[root jlebon]# stapvirt query TestVM
Name: TestVM
UUID: 905951c0-fa4f-409b-079c-c91ddda27028
State: running (ID 5)
Permanent Ports: 0
Hotplugging: not supported
[root jlebon]#
# This tells us that that hotplugging is not supported. In my case, it's because
# libvirt is not new enough. So we'll have to install permanent SystemTap ports
# on the domain. The 'query' command also tells us that there are currently 0
# permanent ports installed.
# Let's try to run stap anyway and see what it would say. This is by the way,
# how you would target your domain using the --remote switch:
[root jlebon]# stap -ve 'probe begin { printf("hello from inside TestVM\n"); exi
t() }' --remote libvirt://TestVM
stapvirt: ERROR: No SystemTap ports detected and hotplugging not available. Try
using the command 'stapvirt port-add TestVM' to add a port.
error receiving hello from stapsh: no reply on remote 'libvirt://TestVM'
[root jlebon]#
# If you're not running as root (and LIBVIRT_DEFAULT_URI is not set), you can
# explicitly specify the libvirt URI this way:
[jlebon ~]$ stap -ve 'probe begin { printf("hello from inside TestVM\n"); exit()
}' --remote libvirt://TestVM/qemu:///system
# Anyway, we can see that stap isn't happy. At least it tells us right away what
# we need to do to add a port:
[root jlebon]# stapvirt port-add TestVM
Added new port org.systemtap.stapsh.0
The domain must be restarted before changes take effect.
[root jlebon]#
# OK, a new port was added. We can verify this by querying again:
[root jlebon]# stapvirt query TestVM
Name: TestVM
UUID: 905951c0-fa4f-409b-079c-c91ddda27028
State: running (ID 5)
Permanent Ports: 1
Hotplugging: not supported
[root jlebon]#
# It now shows that there is one permanent port. We can also use the port-list
# command to see the exact UNIX socket path that QEMU will create:
[root jlebon]# stapvirt port-list TestVM
/var/lib/libvirt/qemu/TestVM.org.systemtap.stapsh.0.sock
[root jlebon]#
# Because the domain is currently running, we need to restart it before changes
# take effect. And this is not just rebooting it, we need to shut it down and
# then start it back up so that libvirt picks up the new port:
[root jlebon]# virsh shutdown TestVM
Domain TestVM is being shutdown
[root jlebon]# virsh start TestVM
Domain TestVM started
[root jlebon]#
# You can verify that the ports have been installed by opening a console (or
# using ssh) on the domain and navigating to /dev/virtio-ports:
[vm ~]$ cd /dev/virtio-ports
[vm virtio-ports]$ ls -l
total 0
lrwxrwxrwx. 1 root root 11 Sep 25 11:34 org.systemtap.stapsh.0 -> ../vport3p1
[vm virtio-ports]$
# OK, so now let's try stap again:
[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM
hello from inside TestVM
[root jlebon]#
# It works! And that's all there is to it!
# OK, what has to be done in the guest side in order to work this dark magic? It
# needs to be configured so that stapsh gets started and respawned to listen on
# the ports. If the guest supports systemd, then we simply need two files to set
# this up:
# 1. The 99-stapsh.rules file, placed in /usr/lib/udev/rules.d
# 2. The stapsh@.service file, placed in /usr/lib/systemd/system
# With this setup, a new stapsh service will be started for each port detected.
# If the guest does not have systemd, then we need two other files set up:
# 1. The 99-stapsh-init.rules file, placed in /usr/lib/udev/rules.d
# 2. The stapshd.init script file, placed in /etc/rc.d/init.d/ (and renamed to
# stapshd). Also need to 'chkconfig stapshd on'.
# EXTRA 1: running on a remote host
# You can specify any valid libvirt URI (see <libvirt.org/uri.html>). This
# includes remote hosts. Here, we connect to localhost, but you get the gist:
[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM/qemu+ssh://root@localhost/system
root@localhost's password:
hello from inside TestVM
[root jlebon]#
# EXTRA 2: hotplugging
# Here, I'm using libvirt v1.1.1, which supports hotplugging virtio-serial ports
# (which is what stapvirt uses). This is what the 'query' command returns:
[root jlebon]# stapvirt query TestVM
Name: TestVM
UUID: 905951c0-fa4f-409b-079c-c91ddda27028
State: running (ID 9)
Permanent Ports: 0
Hotplugging: supported
[root jlebon]#
# And we don't have to use port-add to add a permanent port, we can just
# straight away use stap:
[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM
hello from inside TestVM
[root jlebon]#
# stapvirt takes care of hotplugging the port, opening the connection for stap,
# and unplugging the port when stap is done its thing.