This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Systemtap cannot probe user programs in docker


    Thanks for your ripley.

I just looked into it today.And it turns out that docker's storage driver might be the problem. In my test case, programs put in aufs or overlay2 filesystem can't be probed with uprobe, even if I run it from host instead of docker container. I modify systemtap's runtime to print inode number and it's correct. But if I mount a host's directory to docker containers, such problems are gone. So I guess it has nothing to do with systemtap.

Then I change docker's driver and test again and again, finding that it only works well with vfs driver. (only tested aufs, overlay2 and vfs.)


On 12/23/2017 08:33 PM, Arkady wrote:
Containers run in different namespaces. For example, inside of a
Docker container PIDs start from 1.
I would suggest to hook the Docker kernel module.


On Sat, Dec 23, 2017 at 12:48 PM, lifeibiren@gmail.com
<lifeibiren@gmail.com> wrote:
HI,

     I try to probe user program in docker but it doesn't return anything at
exit. But it works in the host which runs the docker container. When I probe
kernel in docker, it works fine.

     I google for half a day and got nothing. There are also some other guys
who meet the same problems and can't resolve it. Can anybody help? Thanks.


----------------------------Execution----------------------------------
Run docker by: 'docker run -it --privileged fa57ca962f13 /bin/bash'

$ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
WARNING: function _start return probe is blacklisted: keyword at
para-callgraph.stp:24:1
  source: probe $1.return { trace(-1, $$return) }


While in host:
$ stap para-callgraph.stp 'process("test2").function("*")' -c ./test2
WARNING: function _start return probe is blacklisted: keyword at
para-callgraph.stp:24:1
  source: probe $1.return { trace(-1, $$return) }
          ^
      0 test2(22954):->_start
     15 test2(22954): ->__libc_csu_init
     19 test2(22954):  ->_init
     22 test2(22954):  <-_init
     24 test2(22954):  ->frame_dummy
     28 test2(22954):   ->register_tm_clones
     31 test2(22954):   <-register_tm_clones
     32 test2(22954):  <-frame_dummy
     34 test2(22954): <-__libc_csu_init
     36 test2(22954): ->main
    102 test2(22954):  ->dummy
    105 test2(22954):  <-dummy
    107 test2(22954): <-main
    111 test2(22954): ->__do_global_dtors_aux
    114 test2(22954):  ->deregister_tm_clones
    117 test2(22954):  <-deregister_tm_clones
    119 test2(22954): <-__do_global_dtors_aux
    121 test2(22954): ->_fini
    123 test2(22954): <-_fini

-----------------------------Source-------------------------------
$ cat para-callgraph.stp
#! /usr/bin/env stap
function trace(entry_p, extra) {
   %( $# > 1 %? if (tid() in trace) %)
   printf("%s%s%s %s\n",
          thread_indent (entry_p),
          (entry_p>0?"->":"<-"),
          ppfunc (),
          extra)
}
%( $# > 1 %?
global trace
probe $2.call {
   trace[tid()] = 1
}
probe $2.return {
   delete trace[tid()]
}
%)
probe $1.call   { trace(1, $$parms) }
probe $1.return { trace(-1, $$return) }


---------------------------Enviroment-------------------------
Host OS: Ubuntu 16.04 LTS

$ uname -r
4.4.0-98-generic

$ stap -V
Systemtap translator/driver (version 2.9/0.165, Debian version 2.9-2ubuntu2
(xenial))
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 NLS NSS TR1_UNORDERED_MAP

$ docker version
Client:
  Version:      17.09.0-ce
  API version:  1.32
  Go version:   go1.8.3
  Git commit:   afdb6d4
  Built:        Tue Sep 26 22:42:18 2017
  OS/Arch:      linux/amd64

Server:
  Version:      17.09.0-ce
  API version:  1.32 (minimum version 1.12)
  Go version:   go1.8.3
  Git commit:   afdb6d4
  Built:        Tue Sep 26 22:40:56 2017
  OS/Arch:      linux/amd64
  Experimental: false


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]