Re: gdb stack trace unreadable for a pthreads application

[Paul Pluzhnikov <ppluzhnikov at google dot com>]

On Sat, Apr 25, 2009 at 6:56 AM, x x <bapluda@yahoo.com.br> wrote:

> I have this server app that creates a pool of 24 threads. I had a core dump the other day and tried to investigate the cause.

You didn't say whether SIGSEGV happened on the same host on which you
are trying to analyze the core, but all clues indicate that it didn't.

> I can view the stack trace of the thread that caused the segmentation fault, but I cannot view the stack trace of the other threads.

This is a typical symptom of libc.so.6 mismatch between when core was
produced (on target?) and when it is being analyzed (on host).

> The stack trace of the other threads is important for me to determine the cause of the crash. Except for thread 2 these other threads are blocked inside a pthread_mutex_lock call. But the stack trace doesn't make any sense.
> Please view the gdb output in the bottom.
> Should I upgrade my gdb version? Can you help?
>
> Thanks
>
> Ian
>
>
>
> $ uname -a
> Linux ds5632 2.6.22-8-server #1 SMP Thu Jul 12 16:28:57 GMT 2007 i686 GNU/Linux
>
> $lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: ? ?Ubuntu 6.06 LTS
> Release: ? ? ? ?6.06
> Codename: ? ? ? dapper
>
> $gdb mfsrv core.31589
>
> GNU gdb 6.4-debian
> Copyright 2005 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. ?Type "show warranty" for details.
> This GDB was configured as "i486-linux-gnu"...Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
>

So is this:

> Failed to read a valid object file image from memory.
> Core was generated by `./mfsrv mf MundialFoot 1'.
> Program terminated with signal 11, Segmentation fault.
>

And this:

> warning: Can't read pathname for load map: Input/output error.
> Reading symbols from /usr/lib/libstdc++.so.6...done.
> Loaded symbols for /usr/lib/libstdc++.so.6
> Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
> Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
> Reading symbols from /usr/lib/libmysqlclient_r.so.15...done.
> Loaded symbols for /usr/lib/libmysqlclient_r.so.15
> Reading symbols from /usr/lib/libz.so.1...done.
> Loaded symbols for /usr/lib/libz.so.1
> Reading symbols from /lib/tls/i686/cmov/libcrypt.so.1...done.
> Loaded symbols for /lib/tls/i686/cmov/libcrypt.so.1
> Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done.
> Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1
> Reading symbols from /lib/tls/i686/cmov/libm.so.6...done.
> Loaded symbols for /lib/tls/i686/cmov/libm.so.6
> Reading symbols from /lib/libgcc_s.so.1...done.
> Loaded symbols for /lib/libgcc_s.so.1
> Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
> Loaded symbols for /lib/tls/i686/cmov/libc.so.6
> Reading symbols from /lib/ld-linux.so.2...done.
> Loaded symbols for /lib/ld-linux.so.2
> #0 ?0xb7b9c443 in memmove () from /lib/tls/i686/cmov/libc.so.6
> (gdb) info threads

Threads 2 through 24 are blocked in a system call (in kernel VDSO page):

> ?24 process 31589 ?0xffffe410 in ?? ()
> ?23 process 31591 ?0xffffe410 in ?? ()
> ?22 process 31592 ?0xffffe410 in ?? ()
> ?21 process 31593 ?0xffffe410 in ?? ()
> ?20 process 31594 ?0xffffe410 in ?? ()
> ?19 process 31595 ?0xffffe410 in ?? ()
> ?18 process 31600 ?0xffffe410 in ?? ()
> ?17 process 31601 ?0xffffe410 in ?? ()
> ?16 process 31602 ?0xffffe410 in ?? ()
> ?15 process 31603 ?0xffffe410 in ?? ()
> ?14 process 31608 ?0xffffe410 in ?? ()
> ?13 process 31609 ?0xffffe410 in ?? ()
> ?12 process 31610 ?0xffffe410 in ?? ()
> ?11 process 31611 ?0xffffe410 in ?? ()
> ?10 process 31616 ?0xffffe410 in ?? ()
> ?9 process 31617 ?0xffffe410 in ?? ()
> ?8 process 31618 ?0xffffe410 in ?? ()
> ?7 process 31619 ?0xffffe410 in ?? ()
> ?6 process 31624 ?0xffffe410 in ?? ()
> ?5 process 31625 ?0xffffe410 in ?? ()
> ?4 process 31626 ?0xffffe410 in ?? ()
> ?3 process 31632 ?0xffffe410 in ?? ()
> ?2 process 31633 ?0xffffe410 in ?? ()
> * 1 process 31627 ?0xb7b9c443 in memmove () from /lib/tls/i686/cmov/libc.so.6
> (gdb) bt 12
> #0 ?0xb7b9c443 in memmove () from /lib/tls/i686/cmov/libc.so.6
> #1 ?0x08066b62 in IE_Sock::Recv (this=0x8098268, pch=0x8098264 "", cbToRecv=4, iSecs=-1) at IE_Sock.cc:284
> #2 ?0x08066d51 in IE_Sock::RecvInt (this=0x8098268, pi=0x8098264, iSecs=-1) at IE_Sock.cc:257
> #3 ?0x08070b3f in Thread::InitConnection (this=0x8097dfc) at commands.cc:45
> #4 ?0x08062d73 in Thread::Start (this=0x8097dfc) at SMthreads.cc:195
> #5 ?0x08062ec7 in Thread::Thread_Start (pv=0x8097dfc) at SMthreads.cc:37
> #6 ?0xb7e3f341 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #7 ?0xb7bfa4ee in clone () from /lib/tls/i686/cmov/libc.so.6
> (gdb) thread 2
> [Switching to thread 2 (process 31633)]#0 ?0xffffe410 in ?? ()
> (gdb) bt 12

Typical stack trace when libc.so.6 and libpthread.so.0 are different
at core dump and at analysis time:

> #0 ?0xffffe410 in ?? ()
> #1 ?0xacaf8258 in ?? ()
> #2 ?0xb7e493b4 in ?? () from /lib/tls/i686/cmov/libpthread.so.0
> #3 ?0xacaf8220 in ?? ()
> #4 ?0xb7e44778 in accept () from /lib/tls/i686/cmov/libpthread.so.0
> #5 ?0x080661fd in IE_SockSrv::Accept (this=0x8099530, psock=0x8099100, sz=0x0, sl=0) at IE_Sock.cc:74
> #6 ?0x0808039c in admin_thread::thread_start (pv=0x0) at admin_thread.cc:44
> #7 ?0xb7e3f341 in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #8 ?0xb7bfa4ee in clone () from /lib/tls/i686/cmov/libc.so.6
> (gdb) thread 3
> [Switching to thread 3 (process 31632)]#0 ?0xffffe410 in ?? ()
> (gdb) bt 12
> #0 ?0xffffe410 in ?? ()
> #1 ?0xad2f940c in ?? ()
> #2 ?0x00000021 in ?? ()
> #3 ?0x00000000 in ?? ()
>
> and all remaining the threads (4 to 24) have the same screwed up stack
>
>
>
> ? ? ?Veja quais são os assuntos do momento no Yahoo! +Buscados
> http://br.maisbuscados.yahoo.com
>

Assuming my guess of host/target mismatch is correct, the solution is:

gdb mfsrv
(gdb) set solib-absolute-prefix /path/to/target/root
    # libc.so.6, libpthread.so.0 should be in
/path/to/target/root/lib/tls/i686/cmov
(gdb) core core.31589

Cheers,
-- 
Paul Pluzhnikov