Re: Recent aarch64 kprobes and uprobes patch systemtap testing

[William Cohen <wcohen at redhat dot com>]

On 12/10/2015 04:12 PM, David Long wrote:
> On 12/10/2015 03:24 PM, William Cohen wrote:
>> Hi All,
>>
>> Dave Long and Pratyush Anand have been working on kprobe and uprobe patches for aarch64.  I have built a local version the uprobe/upstream_arm64_devel branch of https://github.com/pratyushanand/linux which includes those patches in a linux-4.4.0-rc3 kernel.
>>
>> The tests seemed to run fairly well and the results have been uploaded to dejazilla:
>>
>> https://web.elastic.org/~dejazilla/viewsummary.php?summary=%3D%27%3C56698DCC.3090207%40redhat.com%3E%27
>>
>>         === systemtap Summary ===
>>
>> # of expected passes        6096
>> # of unexpected failures    111
>> # of unexpected successes    2
>> # of expected failures        333
>> # of unknown successes        2
>> # of known failures        89
>> # of untested testcases        97
>> # of unsupported tests        27
>> runtest completed at Thu Dec 10 05:54:32 2015
>>
>>
>> There are still some areas needing for for aarch64 such as stack backtrace support.
>>
>> The following failure looks suspect because the child process died:
>>
>>
>> spawn stap -g ./systemtap.examples/process/threadstacks.stp -Gsize=65536 -c /root/systemtap_write/systemtap/testsuite/pthread_stacks.x 1024 0 -d /root/systemtap_write/systemtap/testsuite/pthread_stacks.x
>>
>> pthread_stacks.x: ./systemtap.base/pthread_stacks.c:67: main: Assertion `rc == 0' failed.
>>
>> WARNING: Child process exited with signal 6 (Aborted)
>>
>> pthread_stacks.[3567] overwrote __default_stacksize@0x3ffb3be4338 (8388608->65536)
>>
>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>>
>> Pass 5: run failed.  [man error::pass5]
>>
>> FAIL: pthread_stacks -Gsize (0 0)
>>
>> The fslatency-nd and fsslower-nd tests need further investigation:
>>
>> PASS: ./systemtap.examples/lwtools/fslatency-nd build
>> meta taglines 'test_installcheck: stap fslatency-nd.stp 1 1' tag 'test_installcheck' value 'stap fslatency-nd.stp 1 1'
>> attempting command stap fslatency-nd.stp 1 1
>> OUT ERROR: read fault [man error::fault] at 0x0000000000000034 (addr) near operator '@cast' at fslatency-nd.stp:66:15
>> Tracing FS sync reads and writes... Output every 1 secs.
>> WARNING: Number of errors: 1, skipped probes: 1
>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>> Pass 5: run failed.  [man error::pass5]
>> child process exited abnormally
>> RC 1
>> FAIL: ./systemtap.examples/lwtools/fslatency-nd run
>>
>> PASS: ./systemtap.examples/lwtools/fsslower-nd build
>> meta taglines 'test_installcheck: stap fsslower-nd.stp -c "sleep 1"' tag 'test_installcheck' value 'stap fsslower-nd.stp -c "sleep 1"'
>> attempting command stap fsslower-nd.stp -c "sleep 1"
>> OUT ERROR: read fault [man error::fault] at 0x0000000000000034 (addr) near operator '@cast' at fsslower-nd.stp:68:15
>> Tracing FS sync reads and writes slower than 10 ms... Hit Ctrl-C to end.
>> TIME     PID    COMM             FUNC           SIZE     LAT(ms)
>> WARNING: Number of errors: 1, skipped probes: 1
>> WARNING: /root/systemtap_write/install/bin/staprun exited with status: 1
>> Pass 5: run failed.  [man error::pass5]
>> child process exited abnormally
>> RC 1
>> FAIL: ./systemtap.examples/lwtools/fsslower-nd run
>>
>>
>> Also a number of network tests failed like the following
>>
>>
>> TEST PWD=/root/systemtap_write/systemtap/testsuite/systemtap.examples/network
>> meta taglines 'test_check: stap -g -p4 netfilter_drop.stp TCP 1' tag 'test_check' value 'stap -g -p4 netfilter_drop.stp TCP 1'
>> attempting command stap -g -p4 netfilter_drop.stp TCP 1
>> OUT /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2731:1: error: initialization from incompatible pointer type [-Werror]
>>   .hook = enter_netfilter_probe_0,
>>   ^
>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2731:1: error: (near initialization for 'netfilter_opts_0.hook') [-Werror]
>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: unknown field 'owner' specified in initializer
>>   .owner = THIS_MODULE,
>>   ^
>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: initialization from incompatible pointer type [-Werror]
>> /tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.c:2732:1: error: (near initialization for 'netfilter_opts_0.dev') [-Werror]
>> cc1: all warnings being treated as errors
>> make[4]: *** [/tmp/stapbIEqFl/stap_0c0db8521e3ea3b2d26dcde208a77baf_21082_src.o] Error 1
>> make[3]: *** [_module_/tmp/stapbIEqFl] Error 2
>> WARNING: kbuild exited with status: 2
>> Pass 4: compilation failed.  [man error::pass4]
>> child process exited abnormally
>> RC 1
>> FAIL: ./systemtap.examples/network/netfilter_drop build
>>
>> -Will
> 
> 
> Cool. Wish I could make sense of systemtap error messages.

Hi Dave,

This was a data dump, so I haven't made sense of some of it either. :)  The ".hook=..." and ".owner=..." are problems in the systemtap code generation for newer kernel and don't concern the aarch64 kprobes/uprobes work.  The read faults for fslatency-nd.stp and fsslower-nd,stp need to be check more carefully. but they are likely issues with systemtap (they do work on linux-4.2.0 on x86_64).

The "FAIL: pthread_stacks -Gsize (0 0)" looks like it could be an issue with uprobes affecting the running of the program.  Pratyush are you able to run this systemtap test locally?

> 
>  At Will Deacon's suggested I tested probing the instruction in __copy_to_user that can cause a captured kernel exception when an application passes in a bad buffer address.  Unfortunately the result was a hang.  So copy_to/from user is going to have to be blacklisted for now, unless there turns out to be a simple fix. I'm worried there might be other places in the kernel where an otherwise probeable instruction might be expected to generate an exception.
> 
> -dl

So the problem is the issue of nested exceptions in the single step debug exception?  Are there other place in the kernel where similar exceptions could occur, such as memory management code probing an address to verify it is valid?

-Will