This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[Bug runtime/20148] New: kernel BUGs while doing parallel testing


https://sourceware.org/bugzilla/show_bug.cgi?id=20148

            Bug ID: 20148
           Summary: kernel BUGs while doing parallel testing
           Product: systemtap
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: runtime
          Assignee: systemtap at sourceware dot org
          Reporter: dsmith at redhat dot com
  Target Milestone: ---

While doing a parallel testsuite run on the debug RHEL7 ppc64 kernel
(3.10.0-327.el7.ppc64.debug), I see the following kernel BUGs (a mix of
"sleeping function called from invalid context" and "scheduling while atomic"):

====
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: sleeping
function called from invalid context at kernel/mutex.c:576
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: in_atomic(): 1,
irqs_disabled(): 0, pid: 27280, name: stapio
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G           OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac77f0] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac78c0] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7930] [c000000000136364] .__might_sleep+0x1b4/0x2c0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79c0] [c000000000a1199c] .mutex_lock_nested+0x6c/0x5b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ad0] [c00000000012ba48] .srcu_notifier_chain_register+0x38/0x120
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b50] [c0000000007c5240] .cpufreq_register_notifier+0xa0/0xc0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7bc0] [d0000000020a8c54] .systemtap_module_init+0x394/0xad0
[s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9680] ._stp_ctl_write_cmd+0x2f0/0x850 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000000a188] system_call+0x38/0xb4
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: s_27280:
systemtap: 3.1/0.163, base: d000000001fc0000, memory:
755data/942text/721ctx/2058net/1358alloc kb, probes: 1026
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE) nd_s_26825(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded:
stap_e6932f2ea1147417c7915513a7de4ea9_26893]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G           OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7290] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7360] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac73d0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7450] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7570] [c000000000a14fb4]
.schedule_hrtimeout_range_clock+0x144/0x550
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76a0] [c000000000372ab0] .poll_schedule_timeout+0x50/0x130
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7720] [c00000000037372c] .do_select+0x7cc/0xa20
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b20] [c000000000373dec] .core_sys_select+0x46c/0x5b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d10] [c000000000374398] .SyS_pselect6+0x198/0x3a0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE) nd_s_26825(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded:
stap_e6932f2ea1147417c7915513a7de4ea9_26893]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7590] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7660] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7750] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79b0] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a70] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a83ac] ._stp_cleanup_and_exit+0x16c/0x4e0
[s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE) tun sg
pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 11 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7560] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7630] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76a0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7720] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7840] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7980] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a40] [c00000000010bd60] .flush_workqueue+0x190/0x640
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a83f0] ._stp_cleanup_and_exit+0x1b0/0x4e0
[s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE) tun sg
pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 11 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7590] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7660] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7750] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79b0] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a70] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a83f8] ._stp_cleanup_and_exit+0x1b8/0x4e0
[s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE) tun sg
pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7460] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7530] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac75a0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7620] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7740] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7880] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7940] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a30] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ab0] [c000000000a23f9c] .unregister_kprobes.part.14+0x8c/0x160
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b50] [d000000001fd6fcc]
.stapkp_batch_unregister_probes.constprop.615+0x8c/0x1a0 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a841c] ._stp_cleanup_and_exit+0x1dc/0x4e0
[s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:44 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: nd_s_27558:
systemtap: 3.1/0.163, base: d000000003000000, memory:
808data/916text/1021ctx/2058net/13150alloc kb, probes: 1023
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7440] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7510] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7580] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7600] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7720] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7860] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7920] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a10] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a90] [c000000000a24380] .unregister_kretprobes.part.16+0xb0/0x2b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b50] [d000000001fd702c]
.stapkp_batch_unregister_probes.constprop.615+0xec/0x1a0 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a841c] ._stp_cleanup_and_exit+0x1dc/0x4e0
[s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: sleeping
function called from invalid context at kernel/mutex.c:576
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: in_atomic(): 1,
irqs_disabled(): 0, pid: 27280, name: stapio
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 11 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76a0] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7770] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac77e0] [c000000000136364] .__might_sleep+0x1b4/0x2c0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a1199c] .mutex_lock_nested+0x6c/0x5b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7980] [c000000000a22b70] .__free_insn_slot+0x30/0x1a0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a10] [c000000000a1c750] .arch_remove_kprobe+0x30/0x50
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a90] [c000000000a24504]
.unregister_kretprobes.part.16+0x234/0x2b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b50] [d000000001fd702c]
.stapkp_batch_unregister_probes.constprop.615+0xec/0x1a0 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a841c] ._stp_cleanup_and_exit+0x1dc/0x4e0
[s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x10000003
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 11 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7510] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac75e0] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7650] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac77f0] [c000000000a16dc0] ._cond_resched+0x50/0x70
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a119a4] .mutex_lock_nested+0x74/0x5b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7980] [c000000000a22b70] .__free_insn_slot+0x30/0x1a0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a10] [c000000000a1c750] .arch_remove_kprobe+0x30/0x50
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a90] [c000000000a24504]
.unregister_kretprobes.part.16+0x234/0x2b0
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b50] [d000000001fd702c]
.stapkp_batch_unregister_probes.constprop.615+0xec/0x1a0 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a841c] ._stp_cleanup_and_exit+0x1dc/0x4e0
[s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:45 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7590] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7660] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7750] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79b0] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a70] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a8474] ._stp_cleanup_and_exit+0x234/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) stap_f7681184a1bc14585a20ca3db6d3522_27025(OE)
tun sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ibmvscsi
scsi_transport_srp scsi_tgt ibmveth rsxx [last unloaded: nd_s_26825]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7740] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7810] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7880] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7900] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a20] [c000000000a10ac0] .schedule_timeout+0x1f0/0x480
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c0000000000f30ec] .msleep+0x4c/0x70
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a84d4] ._stp_cleanup_and_exit+0x294/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) tun sg pseries_rng nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common ibmvscsi scsi_transport_srp scsi_tgt ibmveth rsxx [last
unloaded: stap_f7681184a1bc14585a20ca3db6d3522_27025]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: irq event
stamp: 0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: hardirqs last 
enabled at (0): [<          (null)>]           (null)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: hardirqs last
disabled at (0): [<c0000000000ce5dc>] .copy_process.part.22+0x5dc/0x1040
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: softirqs last 
enabled at (0): [<c0000000000ce5dc>] .copy_process.part.22+0x5dc/0x1040
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: softirqs last
disabled at (0): [<          (null)>]           (null)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7900] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79d0] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a40] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ac0] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a84dc] ._stp_cleanup_and_exit+0x29c/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) tun sg pseries_rng nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common ibmvscsi scsi_transport_srp scsi_tgt ibmveth rsxx [last
unloaded: stap_f7681184a1bc14585a20ca3db6d3522_27025]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7590] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7660] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7750] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7870] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79b0] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a70] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a8628] ._stp_cleanup_and_exit+0x3e8/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) tun sg pseries_rng nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common ibmvscsi scsi_transport_srp scsi_tgt ibmveth rsxx [last
unloaded: stap_f7681184a1bc14585a20ca3db6d3522_27025]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 9 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac74f0] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac75c0] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7630] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76b0] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac77d0] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7910] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79d0] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ac0] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b40] [d000000001fc8138] ._stp_runtime_contexts_free+0x128/0x1f0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a8638] ._stp_cleanup_and_exit+0x3f8/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: sleeping
function called from invalid context at kernel/mutex.c:576
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: in_atomic(): 1,
irqs_disabled(): 0, pid: 27280, name: stapio
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7770] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7840] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac78b0] [c000000000136364] .__might_sleep+0x1b4/0x2c0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7940] [c000000000a1199c] .mutex_lock_nested+0x6c/0x5b0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7a50] [c00000000012bb68]
.srcu_notifier_chain_unregister+0x38/0x130
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ad0] [c0000000007c52f0] .cpufreq_unregister_notifier+0x90/0xb0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b40] [d000000001fc8384] ._stp_kill_time+0x184/0x190 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a863c] ._stp_cleanup_and_exit+0x3fc/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: BUG: scheduling
while atomic: stapio/27280/0x00000003
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: INFO: lockdep
is turned off.
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) tun sg pseries_rng nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common ibmvscsi scsi_transport_srp scsi_tgt ibmveth rsxx [last
unloaded: stap_f7681184a1bc14585a20ca3db6d3522_27025]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7510] [c000000000019690] .show_stack+0x80/0x380 (unreliable)
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac75e0] [c000000000a36980] .dump_stack+0x28/0x3c
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7650] [c000000000a2abb8] .__schedule_bug+0x78/0xa0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac76d0] [c000000000a16b80] .__schedule+0xbc0/0xd10
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac77f0] [c000000000a10bf4] .schedule_timeout+0x324/0x480
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7930] [c000000000a17164] .wait_for_completion+0x114/0x1b0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac79f0] [c00000000011a3c0] .wait_rcu_gp+0x80/0xb0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7ae0] [c00000000020d668] .synchronize_sched+0xb8/0x170
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7b60] [c00000000012b4c4]
.atomic_notifier_chain_unregister+0xb4/0xe0
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7be0] [d0000000020a836c] ._stp_cleanup_and_exit+0x12c/0x4e0
[s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7cc0] [d0000000020a9968] ._stp_ctl_write_cmd+0x5d8/0x850 [s_27280]
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7d80] [c00000000034cf6c] .SyS_write+0x14c/0x400
May 25 16:03:46 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
[c000000183ac7e30] [c00000000004f050] .kretprobe_trampoline_holder+0x0/0x10
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Page fault in
user mode with in_atomic() = 1 mm = c000000036af3a80
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: NIP =
3fffae52d23c  MSR = 800000000280f032
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Oops: Weird
page fault, sig: 11 [#1]
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: SMP
NR_CPUS=2048 NUMA pSeries
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Modules linked
in: nd_s_27558(OE) s_27280(OE) tun sg pseries_rng nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
crct10dif_common ibmvscsi scsi_transport_srp scsi_tgt ibmveth rsxx [last
unloaded: stap_f7681184a1bc14585a20ca3db6d3522_27025]
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CPU: 10 PID:
27280 Comm: stapio Tainted: G        W  OE  ------------  
3.10.0-327.el7.ppc64.debug #1
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: task:
c00000018354d7e0 ti: c000000183ac4000 task.ti: c000000183ac4000
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: NIP:
00003fffae52d23c LR: 00003fffae52d200 CTR: 0000000000000000
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: REGS:
c000000183ac7ea0 TRAP: 0301   Tainted: G        W  OE  ------------   
(3.10.0-327.el7.ppc64.debug)
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: MSR:
800000000280f032 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI>  CR: 44002824  XER: 00000000
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: CFAR:
00003fffae44f540 DAR: 00003fffad31f4e0 DSISR: 42000000 SOFTE: 1
                                                                GPR00:
000000000000005a 00003fffe7b36160 00003fffae557f70 00003fffacb20000
                                                                GPR04:
0000000000800000 0000000000000003 0000000000020022 ffffffffffffffff
                                                                GPR08:
0000000000000000 0000000000006a90 00003fffad31f2e0 0000000000000000
                                                                GPR12:
0000000000000000 00003fffae5db170
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: NIP
[00003fffae52d23c] 0x3fffae52d23c
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: LR
[00003fffae52d200] 0x3fffae52d200
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: Call Trace:
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: ---[ end trace
ca965e2c24bd7516 ]---
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel:
May 25 16:03:47 ibm-p740-01-lp5.rhts.eng.bos.redhat.com kernel: note:
stapio[27280] exited with preempt_count 2
====

I'd guess that there is some contention when several scripts are simultaneously
starting and exiting.

-- 
You are receiving this mail because:
You are the assignee for the bug.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]