This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

How to use the new upcoming stapvirt binary for VM probing


Hello everyone,

We've been working on a new feature of SystemTap which will
allow users to execute scripts within virtual machines managed
by libvirt. Put briefly, this would allow you to do this from
the host:

[jlebon ~]$ stap script.stp --remote=libvirt://MyVM

This does not use ssh, but a virtio-serial channel, which means
that the VM can have its networking disabled. The virtio-serial
channel is also slightly faster than ssh in most regards for
large amounts of data.

I recently put up a mini-session on the related PR (13078)
demonstrating how it will be used. Here it is inlined for your
viewing pleasure. :) Feedback is always welcome.

Cheers,

Jonathan

--------------------------------------------------------------


# This console session explains how one would use stapvirt and the new
# libvirt:// scheme to execute SystemTap scripts inside virtual machines managed
# by libvirt.

# First I'll add my custom stap install to the path

[jlebon ~]$ PATH=codebase/systemtap/install/bin:$PATH
[jlebon ~]$

# Let's run stapvirt

[jlebon ~]$ stapvirt
stapvirt v2.4
Usage: stapvirt [-c URI] [-d PATH] [-v] COMMAND ARGUMENTS

     -c    Specify the libvirt driver URI to which to connect [default: NULL]
     -d    Specify the directory in which sockets should be placed [default:
           /var/lib/libvirt/qemu]
     -v    Increase verbosity

Try the 'help' command for more information.
[jlebon ~]$

# OK, let's try the 'help' command to see what it can do

[jlebon ~]$ stapvirt help
stapvirt v2.4
Usage: stapvirt [-c URI] [-d PATH] [-v] COMMAND ARGUMENTS

     -c    Specify the libvirt driver URI to which to connect [default: NULL]
     -d    Specify the directory in which sockets should be placed [default:
           /var/lib/libvirt/qemu]
     -v    Increase verbosity

Available commands are:

     help
           Display this message.
     list
           List available domains.
     port-add <domain>
           Add a permanent SystemTap port to the domain's definition. If the
           domain is currently running, it must be restarted before changes take
           effect.
     port-list <domain>
           List the UNIX socket paths of the permanent SystemTap ports in the
           domain's definition.
     port-remove <domain>
           Remove a permanent SystemTap port from the domain's definition. If
           the domain is currently running, it must be restarted before changes
           take effect.
     query <domain>
           Display the following information about the domain: its name, its
           UUID, its state, the number of permanent SystemTap ports installed,
           and whether hotplugging is supported.

Domains can be specified using their name, UUID, or ID.

[jlebon ~]$

# Sweet! Let's see what VMs (aka domains) we have on this machine

[jlebon ~]$ stapvirt list
Available domains on URI 'qemu:///session':
UUID	Name	State	Type
[jlebon ~]$

# Right, the default libvirt URI is NULL, which means that on e.g. f19, it will
# default to qemu:///session. This is the same as virsh:

[jlebon ~]$ virsh list --all
 Id    Name                           State
----------------------------------------------------

[jlebon ~]$

# Most likely, your domains reside on qemu:///system:

[jlebon ~]$ virsh -c 'qemu:///system' list --all
# (You get authenticated)
 Id    Name                           State
----------------------------------------------------
 5     TestVM                         running

[jlebon ~]$ 

# Let's try the same thing with stapvirt

[jlebon ~]$ stapvirt -c 'qemu:///system' list
Available domains on URI 'qemu:///system':
UUID	Name	State	Type
905951c0-fa4f-409b-079c-c91ddda27028	TestVM	running (ID 2)	persistent
[jlebon ~]$ 

# Rather than repeating the -c command everytime, you can either set the
# LIBVIRT_DEFAULT_URI env var to 'qemu:///system', or run everything as root.
# Let's do the latter so that we also don't have to authenticate:

[jlebon ~]$ su
Password:
[root jlebon]#

# OK, now let's try the query command to see more details about TestVM

[root jlebon]# stapvirt query TestVM

              Name:  TestVM
              UUID:  905951c0-fa4f-409b-079c-c91ddda27028
             State:  running (ID 5)
   Permanent Ports:  0
       Hotplugging:  not supported

[root jlebon]# 

# This tells us that that hotplugging is not supported. In my case, it's because
# libvirt is not new enough. So we'll have to install permanent SystemTap ports
# on the domain. The 'query' command also tells us that there are currently 0
# permanent ports installed.

# Let's try to run stap anyway and see what it would say. This is by the way,
# how you would target your domain using the --remote switch:

[root jlebon]# stap -ve 'probe begin { printf("hello from inside TestVM\n"); exi
t() }' --remote libvirt://TestVM
stapvirt: ERROR: No SystemTap ports detected and hotplugging not available. Try 
using the command 'stapvirt port-add TestVM' to add a port.
error receiving hello from stapsh: no reply on remote 'libvirt://TestVM'
[root jlebon]# 

# If you're not running as root (and LIBVIRT_DEFAULT_URI is not set), you can
# explicitly specify the libvirt URI this way:

[jlebon ~]$ stap -ve 'probe begin { printf("hello from inside TestVM\n"); exit()
 }' --remote libvirt://TestVM/qemu:///system

# Anyway, we can see that stap isn't happy. At least it tells us right away what
# we need to do to add a port:

[root jlebon]# stapvirt port-add TestVM
Added new port org.systemtap.stapsh.0
The domain must be restarted before changes take effect.
[root jlebon]#

# OK, a new port was added. We can verify this by querying again:

[root jlebon]# stapvirt query TestVM

              Name:  TestVM
              UUID:  905951c0-fa4f-409b-079c-c91ddda27028
             State:  running (ID 5)
   Permanent Ports:  1
       Hotplugging:  not supported

[root jlebon]# 

# It now shows that there is one permanent port. We can also use the port-list
# command to see the exact UNIX socket path that QEMU will create:

[root jlebon]# stapvirt port-list TestVM
/var/lib/libvirt/qemu/TestVM.org.systemtap.stapsh.0.sock
[root jlebon]# 

# Because the domain is currently running, we need to restart it before changes
# take effect. And this is not just rebooting it, we need to shut it down and
# then start it back up so that libvirt picks up the new port:

[root jlebon]# virsh shutdown TestVM
Domain TestVM is being shutdown

[root jlebon]# virsh start TestVM
Domain TestVM started

[root jlebon]# 

# You can verify that the ports have been installed by opening a console (or
# using ssh) on the domain and navigating to /dev/virtio-ports:

[vm ~]$ cd /dev/virtio-ports
[vm virtio-ports]$ ls -l
total 0
lrwxrwxrwx. 1 root root 11 Sep 25 11:34 org.systemtap.stapsh.0 -> ../vport3p1
[vm virtio-ports]$ 

# OK, so now let's try stap again:

[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM
hello from inside TestVM
[root jlebon]# 

# It works! And that's all there is to it!

# OK, what has to be done in the guest side in order to work this dark magic? It
# needs to be configured so that stapsh gets started and respawned to listen on
# the ports. If the guest supports systemd, then we simply need two files to set
# this up:
#  1. The 99-stapsh.rules file, placed in /usr/lib/udev/rules.d
#  2. The stapsh@.service file, placed in /usr/lib/systemd/system
# With this setup, a new stapsh service will be started for each port detected.
# If the guest does not have systemd, then we need two other files set up:
#  1. The 99-stapsh-init.rules file, placed in /usr/lib/udev/rules.d
#  2. The stapshd.init script file, placed in /etc/rc.d/init.d/ (and renamed to
#     stapshd). Also need to 'chkconfig stapshd on'.

# EXTRA 1: running on a remote host

# You can specify any valid libvirt URI (see <libvirt.org/uri.html>). This
# includes remote hosts. Here, we connect to localhost, but you get the gist:

[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM/qemu+ssh://root@localhost/system
root@localhost's password: 
hello from inside TestVM
[root jlebon]# 

# EXTRA 2: hotplugging

# Here, I'm using libvirt v1.1.1, which supports hotplugging virtio-serial ports
# (which is what stapvirt uses). This is what the 'query' command returns:

[root jlebon]# stapvirt query TestVM

              Name:  TestVM
              UUID:  905951c0-fa4f-409b-079c-c91ddda27028
             State:  running (ID 9)
   Permanent Ports:  0
       Hotplugging:  supported

[root jlebon]# 

# And we don't have to use port-add to add a permanent port, we can just
# straight away use stap:

[root jlebon]# stap -e 'probe begin { printf("hello from inside TestVM\n"); exit
() }' --remote libvirt://TestVM
hello from inside TestVM
[root jlebon]# 

# stapvirt takes care of hotplugging the port, opening the connection for stap,
# and unplugging the port when stap is done its thing.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]