gdb crash when symbol file is modified

[dwk <dwks42 at gmail dot com>]

Hello there,

I'm seeing a crash when gdb has loaded a symbol file and that file is
regenerated on-the-fly while gdb is running. I am generating some code
at runtime in mmap'd regions and then jumping to it. I also generate a
symbols.elf file which is like a detached symbol file (ELF with just
symbol information), and I load this in gdb on-the-fly as my program
runs. The symbol file contains existing symbol names with $new
appended, e.g. "main$new", and none of these names already exist in
other files.

Each time I re-run the program, it will regenerate symbols.elf (though
the contents will be identical). Sometimes I set breakpoints on the
new symbols, and those always turn PENDING when I re-run, even once
the generated code regions get mapped into memory. Worse, however, is
the fact that gdb will often segfault when I re-run. It seems to
segfault about 25% of the time when it's re-loading the symbols.elf
file, seemingly with higher probability if I have a breakpoint set on
one of the symbols.

$ gdb --args gdb --args ./loader ./example
(gdb) r  # run the inner gdb
...
(gdb) add-symbol-file symbols.elf 0
add symbol table from file "symbols.elf" at
        .text_addr = 0x0
(y or n) y
Reading symbols from symbols.elf...(no debugging symbols found)...done.
(gdb) b main$new
Breakpoint 2 at 0x400001ed
(gdb) i b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x0000000007433427
        breakpoint already hit 1 time
2       breakpoint     keep y   0x40000192         <main$new>
(gdb) c
Continuing.
...
[Inferior 1 (process 27354) exited normally]
(gdb) r
`/tmp/symbols.elf' has changed; re-reading symbols.
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
0x00005555558a4d77 in ?? ()
(gdb)  # this is the outer gdb; I pressed CTRL-Z
[8]+  Stopped                 gdb --args gdb --args ./loader ./example
$ file symbols.elf
symbols.elf: ELF 64-bit LSB executable, Intel 80386, version 1 (SYSV),
statically linked, not stripped
$ readelf -s symbols.elf | head

Symbol table '.symtab' contains 3720 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000040000000    74 FUNC    GLOBAL DEFAULT    3
deregister_tm_clones$new
     1: 000000004000004a    90 FUNC    GLOBAL DEFAULT    3
register_tm_clones$new
     2: 00000000400000a4    74 FUNC    GLOBAL DEFAULT    3
__do_global_dtors_aux$new
     3: 00000000400000ee    63 FUNC    GLOBAL DEFAULT    3 frame_dummy$new
     4: 000000004000012d    12 FUNC    GLOBAL DEFAULT    3 __libc_csu_fini$new
     5: 0000000040000139    21 FUNC    GLOBAL DEFAULT    3 _fini$new
     6: 000000004000014e   111 FUNC    GLOBAL DEFAULT    3 __libc_csu_init$new
$ fg
gdb --args gdb --args ./loader ./example


(gdb) bt
#0  0x00005555558a4d77 in ?? ()
#1  0x00005555557b2c2b in ?? ()
#2  0x00005555557a5512 in ?? ()
#3  0x00005555557a5669 in ?? ()
#4  0x00005555557a593e in ?? ()
#5  0x00005555557a6011 in ?? ()
#6  0x00005555557a6383 in ?? ()
#7  0x000055555560ebaf in ?? ()
#8  0x000055555560f861 in ?? ()
#9  0x0000555555735c4b in ?? ()
#10 0x0000555555744c9c in ?? ()
#11 0x0000555555747e84 in ?? ()
#12 0x0000555555748baf in ?? ()
#13 0x00005555556ec6a4 in linux_ptrace_test_ret_to_nx_instr ()
#14 0x00005555556f8565 in linux_ptrace_test_ret_to_nx_instr ()
#15 0x00005555556fb370 in linux_ptrace_test_ret_to_nx_instr ()
#16 0x00005555556e97bd in linux_ptrace_test_ret_to_nx_instr ()
#17 0x000055555576c9ef in ?? ()
#18 0x00005555556fb565 in linux_ptrace_test_ret_to_nx_instr ()
#19 0x000055555573e9f6 in ?? ()
#20 0x000055555574f15f in ?? ()
#21 0x0000555555676f77 in linux_ptrace_test_ret_to_nx_instr ()
#22 0x0000555555847b49 in ?? ()
#23 0x0000555555777fdc in ?? ()
#24 0x00005555557782dd in ?? ()
#25 0x000055555577747a in ?? ()
#26 0x00007ffff7bbcb13 in rl_callback_read_char () from
/lib/x86_64-linux-gnu/libreadline.so.7
#27 0x00005555557773ae in ?? ()
#28 0x0000555555777429 in ?? ()
#29 0x00005555557779ed in ?? ()
#30 0x0000555555776715 in ?? ()
#31 0x0000555555776950 in ?? ()
#32 0x00005555557769c5 in ?? ()
#33 0x000055555576feeb in ?? ()
#34 0x000055555576c9ef in ?? ()
#35 0x00005555557710f6 in ?? ()
#36 0x00005555555ec688 in ?? ()
#37 0x00007ffff59432b1 in __libc_start_main (main=0x5555555ec650,
argc=4, argv=0x7fffffffdc68, init=<optimized out>, fini=<optimized
out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffdc58) at
../csu/libc-start.c:291
#38 0x00005555555ed29a in ?? ()
(gdb) x/5i $rip
=> 0x5555558a4cc3:      movzbl 0x44(%rdi),%eax
   0x5555558a4cc7:      and    $0x18,%eax
   0x5555558a4cca:      cmp    $0x10,%al
   0x5555558a4ccc:      je     0x5555558a4d28
   0x5555558a4cce:      mov    0x40(%rsi),%rax
(gdb) p/x $rdi
    $1 = 0x0
(gdb)

As you can see, I only get a partial stack trace of the gdb crash
(even though I have gdb-dbg installed), but it's clear that gdb is
dereferencing a NULL pointer (probably a NULL pointer to a structure).
I'm not sure whether the problem is caused by a) setting breakpoints
in a memory region that doesn't exist at load-time, b) rewriting the
symbols file at runtime, or c) some combination. While it's true that
the symbols.elf file may be temporarily invalid while I'm writing to
it, gdb only re-parses the file when I type "run" after the program
has successfully exited.

I should perhaps just remove-symbol-file and then add-symbol-file
again each time, this has other side effects such as clearing my
displays. Any comments welcome. I run into this crash at least 5-10
times per day and it's quite frustrating.

-- dwk.