gdb/gdbserver running arm multithreaded applications on old glibc

["Bill Schiller" <wbschiller at gmail dot com>]

I know I'm looking for a bit of charity work but I've run out of
options.  I am in the unenviable position of being locked into using
an ancient toolchain and libc version (due to supporting a product in
the field).  I'm hoping somebody on this list may recall the issue I
describe below and have some tips on debugging and/or resolving the
problem.

I am attempting to use gdb/gdbserver to debug a remote application
running on an arm/xscale target.  Also, my application is very thread
intensive which seems to be adding to my problems.

My development environment is as follows:
  -toolchain from ipaq/familiar project, release 0.5.1, arm-linux-gcc
   2.95.3 provided by debian pacakge cross-arm-toolchain_2.95.3-1_i386.deb
  -glibc 2.2.3 built from:
     glibc-2.2.3-nss.diff
     glibc-2.2.3.tar.gz
     glibc-linuxthreads-2.2.3.tar.gz
  -linux kernel 2.4.18-rmk7-pxa3-cerf1 based on familiar release + patches

I have built a gdbserver for the target and an arm-linux-gdb for cross
debugging using gdb 6.7.1.  These look to be working well for basic
debugging.  Performance is also as expected for some small threaded
applications.

Unfortunately, problems occur when running applications that spawn
multiple threads and perform significant memory allocation.  For
example, the application included below (threadtest.c) runs as
expected on the target.  It even runs as expected inside gdb on the
target.  But when run inside gdbserver, the application runs for a
while and then stops.  The cpu usage is reported as 0 by top.

Below you can see the output of the arm-linux-gdb session.  It was
interrupted when the cpu usage was 0.  There appeared to be no
deadlocks from the stack traces.  After continuing the debugger, the
application eventually exits but it takes much longer than expected
(on the order of minutes instead of seconds).  During the remainder of
the execution, there is little to no cpu usage.  Also, top shows
several zombie processes (for the completed threads) that appear to
stick around until the thread is joined.

>From what I can tell, it appears that gdbserver is ignoring/missing
some signal or event.  Does this issue sound familiar to anyone?  Any
ideas on how to go about debugging this issue?

I believe this problem would go away if I updated the toolchain, glibc
and pthreads.  Unfortunately this isn't an option right now.  I could
add patches by hand to glibc, pthreads and or the kernel if someone
knows of a fix.

Any help is greatly appreciated.

Bill

********** start of threadtest.c **********************
/**
 * Build by: arm-linux-gcc -g threadtest.c -lpthread -o armthreadtest
 */
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#define NUM_THREADS 50

void *thread_function(void *arg)
{
    size_t i;
    char *str = NULL;

    printf("Starting allocation for thread %d.\n", arg);

    for ( i = 0; i < 2*10240; i++ )
    {
        str = (char*) realloc(str, i);
    }
    free(str);

    printf("Finished allocation for thread %d.\n", arg);

    pthread_exit(NULL);
}

int main(int argc, char *argv[])
{
    pthread_t threads[NUM_THREADS];
    int i;

    for (i = 0; i < NUM_THREADS; i++)
    {
        pthread_create(&threads[i], NULL, thread_function, (void *)i);
    }

    printf("Started %d threads.\n", NUM_THREADS);

    for (i = 0; i < NUM_THREADS; i++)
        pthread_join(threads[i], NULL);

    printf("Completed test.\n");

    return 0;
}
********** end of threadtest.c **********************

********** start of arm-linux-gdb output **********************
$ arm-linux-gdb ./armthreadtest
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-linux"...
(gdb) set solib-absolute-prefix /usr/local/arm/2.95.3/arm-linux/
(gdb) set solib-search-path /usr/local/arm/2.95.3/arm-linux/lib
(gdb) target remote 192.168.1.10:40000
Remote debugging using 192.168.1.10:40000
0x40002a90 in _start () from /usr/local/arm/2.95.3/arm-linux/lib/ld-linux.so.2
Current language:  auto; currently c
(gdb) cont
Continuing.
[New Thread 17526]

Program exited normally.
(gdb) target remote 192.168.1.10:40000
Remote debugging using 192.168.1.10:40000
0x40002a90 in _start () from /usr/local/arm/2.95.3/arm-linux/lib/ld-linux.so.2
(gdb) cont
Continuing.
[New Thread 17579]

Program exited normally.
(gdb) target remote 192.168.1.10:40000
Remote debugging using 192.168.1.10:40000
0x40002a90 in _start () from /usr/local/arm/2.95.3/arm-linux/lib/ld-linux.so.2
(gdb) cont
Continuing.
[New Thread 17639]

Program received signal SIGINT, Interrupt.
0x40071b30 in __sigsuspend (set=0xbffffa8c) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
55                                         CHECK_SIGSET (set), _NSIG / 8);
(gdb) info threads
  25 Thread 17696  0x40071b30 in __sigsuspend (set=0xb99ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  24 Thread 17695  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  23 Thread 17694  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  22 Thread 17693  0x40071b30 in __sigsuspend (set=0xb9fffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  21 Thread 17686  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  20 Thread 17685  0x40071b30 in __sigsuspend (set=0xbafffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  19 Thread 17684  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  18 Thread 17683  0x40071b30 in __sigsuspend (set=0xbb3ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  17 Thread 17680  0x40071b30 in __sigsuspend (set=0xbb9ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  16 Thread 17679  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  15 Thread 17677  0x40071b30 in __sigsuspend (set=0xbbfffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  14 Thread 17676  0x40071b30 in __sigsuspend (set=0xbc1ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  13 Thread 17675  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  12 Thread 17673  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  11 Thread 17672  0x40071b30 in __sigsuspend (set=0xbc9ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  10 Thread 17670  0x40071b30 in __sigsuspend (set=0xbcdffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  9 Thread 17669  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  8 Thread 17665  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  7 Thread 17663  0x40071b30 in __sigsuspend (set=0xbdbffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  6 Thread 17660  0x40071b30 in __sigsuspend (set=0xbe1ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  5 Thread 17659  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  4 Thread 17652  0x40071b30 in __sigsuspend (set=0xbf1ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
  3 Thread 17651  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
  2 Thread 17648  0x40105324 in __poll (fds=0x12760, nfds=1,
timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:52
* 1 Thread 17639  0x40071b30 in __sigsuspend (set=0xbffffa8c) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
(gdb) bt
#0  0x40071b30 in __sigsuspend (set=0xbffffa8c) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
#1  0x4002b5b4 in __pthread_wait_for_restart_signal (self=0x4003c140)
at pthread.c:942
#2  0x40028060 in pthread_join (thread_id=3076,
thread_return=<incomplete type>) at restart.h:36
#3  0x000086a4 in main (argc=1, argv=0xbffffcf4) at threadtest.cpp:35
(gdb) thread 2
[Switching to thread 2 (Thread 17648)]#0  0x40105324 in __poll
(fds=0x12760, nfds=1, timeout=2000) at
../sysdeps/unix/sysv/linux/poll.c:52
52            int retval = INLINE_SYSCALL (poll, 3, CHECK_N (fds,
nfds), nfds, timeout);
(gdb) bt
#0  0x40105324 in __poll (fds=0x12760, nfds=1, timeout=2000) at
../sysdeps/unix/sysv/linux/poll.c:52
#1  0x400283fc in __pthread_manager (arg=0xfffffffc) at manager.c:139
#2  0x400285bc in __pthread_manager_event (arg=0x4003c948) at manager.c:221
#3  0x4010cd9c in clone () from /usr/local/arm/2.95.3/arm-linux/lib/libc.so.6
Backtrace stopped: frame did not save the PC
(gdb) thread 3
[Switching to thread 3 (Thread 17651)]#0  pthread_handle_sigrestart
(sig=1073989972) at pthread.c:796
796     {
(gdb) bt
#0  pthread_handle_sigrestart (sig=1073989972) at pthread.c:796
#1  <signal handler called>
#2  0x40071b30 in __sigsuspend (set=0xbf3ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
#3  0x4002b5b4 in __pthread_wait_for_restart_signal (self=0xbf3ffc00)
at pthread.c:942
#4  0x4002d778 in __pthread_alt_lock (lock=<incomplete type>,
self=0xbf3ffc00) at restart.h:36
#5  0x400296c0 in __pthread_mutex_lock (mutex=0x40155454) at mutex.c:120
#6  0x400b5ef4 in __libc_realloc (oldmem=0x40155044, bytes=14853) at
malloc.c:3281
#7  0x000085b4 in thread_function (arg=0x2) at threadtest.cpp:13
#8  0x400286ac in pthread_start_thread (arg=0xfffffffc) at manager.c:274
#9  0x40028704 in pthread_start_thread_event (arg=0xbf3ffc00) at manager.c:298
#10 0x4010cd9c in clone () from /usr/local/arm/2.95.3/arm-linux/lib/libc.so.6
Backtrace stopped: frame did not save the PC
(gdb) thread 4
[Switching to thread 4 (Thread 17652)]#0  0x40071b30 in __sigsuspend
(set=0xbf1ffa10) at ../sysdeps/unix/sysv/linux/sigsuspend.c:55
55                                         CHECK_SIGSET (set), _NSIG / 8);
(gdb) bt
#0  0x40071b30 in __sigsuspend (set=0xbf1ffa10) at
../sysdeps/unix/sysv/linux/sigsuspend.c:55
#1  0x4002b5b4 in __pthread_wait_for_restart_signal (self=0xbf1ffc00)
at pthread.c:942
#2  0x4002d778 in __pthread_alt_lock (lock=<incomplete type>,
self=0xbf1ffc00) at restart.h:36
#3  0x400296c0 in __pthread_mutex_lock (mutex=0x40155454) at mutex.c:120
#4  0x400b5ef4 in __libc_realloc (oldmem=0x40155044, bytes=12108) at
malloc.c:3281
#5  0x000085b4 in thread_function (arg=0x3) at threadtest.cpp:13
#6  0x400286ac in pthread_start_thread (arg=0xfffffffc) at manager.c:274
#7  0x40028704 in pthread_start_thread_event (arg=0xbf1ffc00) at manager.c:298
#8  0x4010cd9c in clone () from /usr/local/arm/2.95.3/arm-linux/lib/libc.so.6
Backtrace stopped: frame did not save the PC
(gdb) cont
Continuing.

Program exited normally.
(gdb)
********** end of arm-linux-gdb output **********************