This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: [Qemu-devel] Using the qemu tracepoints with SystemTap
On Wed, Sep 14, 2011 at 5:02 PM, William Cohen <wcohen@redhat.com> wrote:
> On 09/14/2011 10:51 AM, Stefan Hajnoczi wrote:
>> On Tue, Sep 13, 2011 at 8:38 PM, William Cohen <wcohen@redhat.com> wrote:
>>> On 09/13/2011 12:10 PM, William Cohen wrote:
>>>
>>>> Should the qemu.kvm.cpu_in and qemu.kvm.cpu_out match up? ?There are a lot more qemu.kvm.cpu_out than qemu.kvm.cpu_in count.
>>>
>>> I found that cpu_in and cpu_out refer to input and output instructions. ?I wrote a little script tally up the input and output operations on each port to run on a qemu on fc15 machine.
>>>
>>> It generates output like the following:
>>>
>>>
>>> cpu_in
>>> ?port ? ?count
>>> 0x01f7 ? ? 3000
>>> 0x03d5 ? ? ?120
>>> 0xc000 ? ? 2000
>>> 0xc002 ? ? 3000
>>>
>>> cpu_out
>>> ?port ? ?count
>>> 0x0080 ? ? ?480
>>> 0x01f1 ? ? 2000
>>> 0x01f2 ? ? 2000
>>> 0x01f3 ? ? 2000
>>> 0x01f4 ? ? 2000
>>> 0x01f5 ? ? 2000
>>> 0x01f6 ? ? 2000
>>> 0x01f7 ? ? 1000
>>> 0x03d4 ? ? ?480
>>> 0x03d5 ? ? ?120
>>> 0x03f6 ? ? 1000
>>> 0xc000 ? ? 3000
>>> 0xc002 ? ? 2000
>>> 0xc004 ? ? 1000
>>> 0xc090 ? ? ? ?4
>>>
>>> Looks like lots of touching the ide device ports (0x01f0-0x01ff) and some vga controller (0x03d0-0x3df). This is kind of what would be expected when the machine is doing a fsck and selinux relabel on the guest virtual machines. Look like some pci device access (http://www.tech-pro.net/intro_pci.html) also.
>>
>> I think the cpu_in/cpu_out tracing script can be useful in identifying
>> performance problems, especially when obscure guest OSes experience
>> poor performance due to weird ways of interacting with hardware.
>
> Glad this example looks useful.
>
>> Where are you putting your SystemTap scripts? ?I suggest creating a
>> public git repo and adding a link from the QEMU wiki tracing page:
>> http://wiki.qemu.org/Features/Tracing
>
> Definitely need a more lasting place to put the QEMU examples. SystemTap has an examples directory which try to put things like this into (http://sourceware.org/systemtap/examples/). These are run as part of the testsuite. These qemu examples are not ready for inclusion in the examples. I have set up https://github.com/wcohen/qemu_systemtap for the qemu examples.
Added a link from http://qemu.org/Features/Tracing.
>> Perhaps we could even include them in a contrib/ or similar directory
>> in qemu.git so that distros can ship them. ?But before we worry about
>> that we need a useful set of things that can be observed.
>
> We definitely want to develop scripts that give people a better understanding what is going on. I am just now learning how kvm and qemu work, so the early scripts are just counting events to give people an idea events are frequent. In some cases these could point to issues where some assumptions about event frequency are incorrect. I would like to have scripts that examine the sequences of events and indicate when long latencies occur, but I don't yet have enough understanding of the how kvm and qemu work to implement those.
The vmexit/entry things I mentioned in the mail above are events that
can be traced in the host kernel (kvm:kvm_exit and kvm:kvm_entry).
The QEMU-side equivalent is the ioctl(KVM_RUN) but we do not have any
trace events in kvm-all.c yet (!!). A trace event before the ioctl
and one after could be used to trace vcpu code execution from QEMU's
perspective.
Stefan