[HELP] Systemtap fails to load kernel module

[Jovi Zhangwei <jovi at cloudflare dot com>]

Hi,

Our production systems always reports cannot load kernel module when
using systemtap, the reason seems to be systemtap need large trunk
memory, but the system cannot allocate it due to the memory
fragmentation.

I found one workaround is decrease the value of
/proc/sys/vm/min_free_kbytes, it works, but it's still a workaround,
and it's annoying to change that value again and again.

We also tried to use bulk mode in systemtap, it eliminated the memory
allocation error, but the retrieved output is unreadable, so we
discard the bulk mode.

Note that there have a lot of free memory in system, it's not because
of out of memory, it's memory fragmentation.

We cannot found another solution to this issue, so I come here to ask
help, I'm sure there must have someone encountered this issue before,
as memory fragmentation is so common in production systems. Any
comments is great appreciated.

Trackback:

[747067.765790] staprun: page allocation failure: order:5, mode:0x10c0d0
[747067.774148] CPU: 8 PID: 25371 Comm: staprun Tainted: G           O
3.14.28-cloudflare #1
[747067.784813] Hardware name: HP ProLiant DL180 G6  , BIOS O20 09/01/2011
[747067.792323]  ffff88183fc4deb8 ffffffff81466263 0000000000000001
ffffffff810d7968
[747067.795564]  ffffffe000000000 0000000000000000 0000000000000000
0000000000000000
[747067.795566]  0000000000000020 0000000000000002 0000000000000010
ffff8817fc45aa90
[747067.795569] Call Trace:
[747067.795571]  [<ffffffff81466263>] ? dump_stack+0x41/0x51
[747067.795580]  [<ffffffff810d7968>] ? warn_alloc_failed+0x111/0x138
[747067.795588]  [<ffffffff810dab95>] ? __alloc_pages_nodemask+0x653/0x7f1
[747067.795592]  [<ffffffff8110730f>] ? alloc_pages_current+0xc0/0xda
[747067.795598]  [<ffffffff810d6e1c>] ? __get_free_pages+0xd/0x3a
[747067.795601]  [<ffffffff810eca48>] ? kmalloc_order_trace+0x23/0x7d
[747067.795607]  [<ffffffff810ad216>] ? relay_open+0x51/0x201
[747067.795614]  [<ffffffffa04d95f7>] ?
_stp_transport_data_fs_init+0xfa/0x140
[stap_c7d18892b06f53512b6e2aa501c1830_25371]
[747067.795649]  [<ffffffff811d9d26>] ? __create_file+0x1b3/0x1dc
[747067.795666]  [<ffffffffa04dafa0>] ? init_module+0x2eb/0x8d1
[stap_c7d18892b06f53512b6e2aa501c1830_25371]
[747067.795680]  [<ffffffffa04dacb5>] ?
_stp_pmap_new.constprop.88+0xc2/0xc2
[stap_c7d18892b06f53512b6e2aa501c1830_25371]
[747067.795686]  [<ffffffff8100028e>] ? do_one_initcall+0x78/0xf7
[747067.795691]  [<ffffffff8146c6fd>] ? notifier_call_chain+0x32/0x52
[747067.795697]  [<ffffffff810583ac>] ? __blocking_notifier_call_chain+0x48/0x53
[747067.795702]  [<ffffffff8109431f>] ? load_module+0x181f/0x1dc7
[747067.795706]  [<ffffffff810fb1a3>] ? vmap_page_range_noflush+0x225/0x275
[747067.795709]  [<ffffffff81094953>] ? SyS_init_module+0x8c/0x97
[747067.795711]  [<ffffffff81094953>] ? SyS_init_module+0x8c/0x97

[747067.795715] [<ffffffff8146fa63>] ? tracesys+0xe1/0xe6