How to debug gdbserver/thread SIG32 problem?

[Grant Edwards <grant dot b dot edwards at gmail dot com>]

Can anybody provide any hints on how to troubleshoot debugging
of a threaded app using gdbserver?  When a new thread is
started, I get a SIG32 and everything is pretty muck borked at
that point.

This problem seems to have come up regularly for about 7 years
now.   Some people seem to be able to get it working, so there
must be a secret recipe somewhere.

Gdb seems aware of threads: I see "New Thread" messages as I
should, and "info threads" shows the correct number of threads
with correct PIDs.  But, once a thread is created and the SIG32
has happened, nothing else works right. Here's a session where
I set a breakpoint on the line of code that creates the first
thread.  I then "next" past that line:

   GNU gdb 6.8
   [...]
   This GDB was configured as "--host=i386-pc-linux-gnu --target=arm-linux-uclibc".
   [New Thread 839]
   0x40000930 in _start ()
      from /home/grante/processors/atmel/RM9200/buildroot-2009.05/project_build_arm/uclibc/root/lib/ld-uClibc.so.0
   
   (gdb) break 62
   Breakpoint 1 at 0x8dc8: file hello.c, line 62.
   
   (gdb) c
   Continuing.
   
   Breakpoint 1, main () at hello.c:62
   62            s = pthread_create(tid+i, attrp, thread, (void*)i);  abort_if_error(s);
   
   (gdb) next
   
   Program received signal SIG32, Real-time event 32.
   0x400445d4 in ?? ()
   
   (gdb) info threads
   [New Thread 840]
   [New Thread 841]
     3 Thread 841  0x40065700 in ?? ()
     2 Thread 840  0x40043918 in ?? ()
   * 1 Thread 839  0x400445d4 in ?? ()
   warning: Couldn't restore frame in current thread, at frame 0
   0x400445d4 in ?? ()
   
   (gdb) quit
   The program is running.  Exit anyway? (y or n) EOF [assumed Y]
   
[The output from "gdbserver --debug" for the above session is
below -- it's a bit long]

As you can see above, the dev-host is IA32/linux-glibc and the
target is an ARM920/linux-uclibc

So far I've confirmed that

 1) gdbserver is finding and opening libthread_db.so OK, and I
    don't see any error messages relating to libthread_db.

 2) gdb on the dev-host is finding and opening target
    architecture .so files.  I have solib-absolute-prefix set
    to point to target library files, and I think that the
    message bout being in _start() confirms that it's set
    correctly.
    
 3) Target architecture lib_pthread.so on both target and host
    are not stripped.

 4) In the output from "configure" for gdbserver, it is able to
    find libthread_db.h and "ldd" shows that gdbserver on the
    target is linked against lib_threaddb.so

I've tried using both NPTL and linuxthreads, and saw the same
behavior with both.

One thing I'm suspicious about is that a common bit of advice
seems to be to make sure that libpthread.so isn't stripped (it
isn't).  However, strace doesn't show any attempts to
locate/open libpthread.so by either gdb or gdbserver.  Is that
significant?

Am I right in assuming it's the gdbserver that's broken?  It
doesn't seem like the $vCont response of $T4d sent by gdbserver
(which is what causes the SIG32 message). $T4d is the "wrong
answer" -- correct?

I'd be happy to debug gdbserver if somebody could provide me
with a hint as to what I'm looking for.



-----------------------  gdbserver --debug  ----------------------
# gdbserver --debug 10.0.0.98:12345 /usr/bin/hello
Process /usr/bin/hello created; pid = 839
Got an event from 839 (57f)
stop pc is 40000930
stop pc is 40000930
Hit a non-gdbserver breakpoint.
Listening on port 12345
Remote debugging from host 10.0.0.1
Writing resume reply for 839

Writing 24 to 40000ab0
Writing 01 to 40000ab0
Writing 01 to 00008dc8
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
  stop pc is 40000930
Got an event from 839 (57f)
stop pc is 40000ab0
stop pc is 40000ab0
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 0d to 40000ab0
Writing 1c to 00008dc8
Writing 01 to 40000ab4
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 40000ab0
Removed breakpoint.
  stop pc is 40000ab0
Got an event from 839 (57f)
stop pc is 40000ab4
stop pc is 40000ab4
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 00 to 40000ab4
Writing 01 to 40000ab0
Writing 01 to 00008dc8
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 40000ab4
Removed breakpoint.
  stop pc is 40000ab4
Got an event from 839 (57f)
stop pc is 40000ab0
stop pc is 40000ab0
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 0d to 40000ab0
Writing 1c to 00008dc8
Writing 01 to 40000ab4
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 40000ab0
Removed breakpoint.
  stop pc is 40000ab0
Got an event from 839 (57f)
stop pc is 40000ab4
stop pc is 40000ab4
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 00 to 40000ab4
Writing 01 to 40000ab0
Writing 01 to 00008dc8
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 40000ab4
Removed breakpoint.
  stop pc is 40000ab4
main()
Got an event from 839 (57f)
stop pc is 00008dc8
stop pc is 00008dc8
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 0d to 40000ab0
Writing 1c to 00008dc8
Writing 01 to 00008dcc
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dc8
Removed breakpoint.
  stop pc is 00008dc8
Got an event from 839 (57f)
stop pc is 00008dcc
stop pc is 00008dcc
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 03 to 00008dcc
Writing 01 to 40000ab0
Writing 01 to 00008dc8
Writing 01 to 00008dd0
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dcc
Removed breakpoint.
  stop pc is 00008dcc
Got an event from 839 (57f)
stop pc is 00008dd0
stop pc is 00008dd0
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 9c to 00008dd0
Writing 01 to 00008dd4
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dd0
Removed breakpoint.
  stop pc is 00008dd0
Got an event from 839 (57f)
stop pc is 00008dd4
stop pc is 00008dd4
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 02 to 00008dd4
Writing 01 to 00008dd8
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dd4
Removed breakpoint.
  stop pc is 00008dd4
Got an event from 839 (57f)
stop pc is 00008dd8
stop pc is 00008dd8
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 1c to 00008dd8
Writing 01 to 00008ddc
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dd8
Removed breakpoint.
  stop pc is 00008dd8
Got an event from 839 (57f)
stop pc is 00008ddc
stop pc is 00008ddc
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 03 to 00008ddc
Writing 01 to 00008de0
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008ddc
Removed breakpoint.
  stop pc is 00008ddc
Got an event from 839 (57f)
stop pc is 00008de0
stop pc is 00008de0
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 14 to 00008de0
Writing 01 to 00008de4
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008de0
Removed breakpoint.
  stop pc is 00008de0
Got an event from 839 (57f)
stop pc is 00008de4
stop pc is 00008de4
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing e4 to 00008de4
Writing 01 to 00008de8
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008de4
Removed breakpoint.
  stop pc is 00008de4
Got an event from 839 (57f)
stop pc is 00008de8
stop pc is 00008de8
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 0c to 00008de8
Writing 01 to 00008dec
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008de8
Removed breakpoint.
  stop pc is 00008de8
Got an event from 839 (57f)
stop pc is 00008dec
stop pc is 00008dec
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing cb to 00008dec
Writing 01 to 00008920
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008dec
Removed breakpoint.
  stop pc is 00008dec
Got an event from 839 (57f)
stop pc is 00008920
stop pc is 00008920
Hit a non-gdbserver breakpoint.
Writing resume reply for 839

Writing 00 to 00008920
Writing 01 to 00008df0
Resuming, no pending status
Resuming process 839 (continue, signal 0, stop not expected)
Checking for breakpoint in process 839.
stop pc is 00008920
Removed breakpoint.
  stop pc is 00008920
Got an event from 839 (3057f)
stop pc is 40042454
Resuming process 839 (continue, signal 0, stop not expected)
  stop pc is 40042454
Got an event from 841 (137f)
Got an event from 840 (3057f)
stop pc is 40042454
thread running for port 0 buffer=0x116e8
Server-Using 0.0.0.0 and port 0...
Waiting for connection on port 0
Resuming process 840 (continue, signal 0, stop not expected)
  stop pc is 40042454
stop pc is 400445d4
Sending sigstop to process 840
Sending sigstop to process 841
Got an event from 840 (137f)
stop pc is 40043918
Got an event from 841 (137f)
stop pc is 40065700
Writing resume reply for 839

Writing 00 to 00008df0
Writing 0d to 40000ab0
Writing 1c to 00008dc8
Killing inferior
Got an event from 840 (9)
LWP 840 exiting
Got an event from 841 (9)
LWP 841 exiting
Got an event from 839 (9)
LWP 839 exiting
------------------------------------------------------------------