Re: Segmentation Fault: Task died at uprobe probepoint

[Mark Wielaard <mjw at redhat dot com>]

On Mon, 2010-06-14 at 20:52 -0700, Gaurav Dhiman wrote:
> On Mon, Jun 14, 2010 at 6:33 PM, Frank Ch. Eigler <fche@redhat.com> wrote:
> > Gaurav Dhiman <dimanuec@gmail.com> writes:
> >> [...]
> >> Following are the logs from the Pass 5 of the run with stap -vvvt
> >> -DDEBUG_UPROBES:
> >>
> >> Pass 5: starting run.
> >> [...]
> >> stapio:stp_main_loop:571 probe_start() returned 0
> >> stap_uprobe_change_plus:67: +uprobe spec 0 idx 0 process uprobe-test[25763] addr 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
> >> test begin!!
> >> test_func called!
> >> i=0x5
> >> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process uprobe-test[25763] reloc 0000000000400517 pp process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
> >>
> >> The program receives a SIGSEGV signal at this point.
> >
> > OK, something is wrong here.  See that -uprobe line?  It means stap
> > has for some reason determined that it's time to remove the uprobe,
> > but that should not happen until either the script or the target
> > program exits.
> >
> > This will make things more verbose, but could you try adding
> > -DDEBUG_TASK_FINDER and -DDEBUG_TASK_FINDER_VMA to the stap
> > command line?
> >
> You are right, that looks suspicious; it is actually removing the
> probe, when it should not be doing it. Here is the more verbose
> output:
> 
> test_func called!
> i=0x5
> stap_uprobe_process_found:246: -proc pid 27237 stf ffffffffa001dce0
> ffffffffa001dce0 path /home/gdhiman/tests/uprobe-test
> stap_uprobe_change_minus:220: -uprobe spec 0 idx 0 process
> uprobe-test[27237] reloc 0000000000400517 pp
> process("/home/gdhiman/tests/uprobe-test").function("test_func@/home/gdhiman/tests/uprobe-test.c:4")
> _stp_tf_exec_cb:29: tsk 27237:27237 , register_p: 0, process_p: 1

Unless I am reading the logs wrong, I do think this is expected. The
process gets a SIGSEGV and so uprobes is notified that it needs to clean
up (-proc pid 27237 means process is gone).

> > I meant the uprobe-test binary, just to see what instructions were
> > in the vicinity of the test_func function.  Just that area.
> >
> 
> Here it is:
> 
> 000000000040050c <test_func>:
>   40050c:       55                      push   %rbp
>   40050d:       48 89 e5                mov    %rsp,%rbp
>   400510:       48 83 ec 10             sub    $0x10,%rsp
>   400514:       89 7d fc                mov    %edi,-0x4(%rbp)
>   400517:       8b 75 fc                mov    -0x4(%rbp),%esi
>   40051a:       bf 3c 06 40 00          mov    $0x40063c,%edi
>   40051f:       b8 00 00 00 00          mov    $0x0,%eax
>   400524:       e8 cf fe ff ff          callq  4003f8 <printf@plt>
>   400529:       8b 45 fc                mov    -0x4(%rbp),%eax
>   40052c:       c9                      leaveq
>   40052d:       c3                      retq

That does look OK, even though it looks different from my local build:

Dump of assembler code for function test_func:
   0x080483d0 <+0>:	push   %ebp
   0x080483d1 <+1>:	mov    %esp,%ebp
   0x080483d3 <+3>:	push   %ebx
   0x080483d4 <+4>:	sub    $0x14,%esp
   0x080483d7 <+7>:	mov    0x8(%ebp),%ebx
   0x080483da <+10>:	movl   $0x80484f4,(%esp)
   0x080483e1 <+17>:	mov    %ebx,0x4(%esp)
   0x080483e5 <+21>:	call   0x80482f4 <printf@plt>
   0x080483ea <+26>:	mov    %ebx,%eax
   0x080483ec <+28>:	add    $0x14,%esp
   0x080483ef <+31>:	pop    %ebx
   0x080483f0 <+32>:	pop    %ebp
   0x080483f1 <+33>:	ret    
End of assembler dump.

What compiler and flags did you use? Here gcc version 4.4.4 20100503
(Red Hat 4.4.4-2) with -g -O2.

If possible could you set ulimit -c unlimited and catch the core file of
the program and give the disassembly from test_func in the core? Maybe
the breakpoint instruction in there was misplaced?

Thanks,

Mark