[RFC][PATCH] Demangler crash resistance

[Gary Benson <gbenson at redhat dot com>]

Hi all,

A number of demangler crashes got filed recently, at least some of
which are likely my fault.  These need fixing, of course, but this
highlighted to me the fact that GDB is probably the most sensitive
user of libiberty's demangler in terms of crashes, etc.  GCC and glibc
only enter the demangler for error conditions, whereas GDB demangles
symbols in general use.  Furthermore, a given GDB session will likely
demangle a lot of symbols.  On the other side of the coin, I don't
think many GDB hackers have the depth of knowledge to fix some of the
more complicated crashes, in terms of understanding what the demangler
is supposed to be producing.  Feel free to correct me here (but be
prepared for some questions if you do!)

Here is a quandry where the people with the expertise to fix demangler
crashes chiefly work on projects where such bugs are low priority.
They're high priority for GDB, but we don't have the expertise to fix
them.  While the bugs are unfixed, our users may not be able to use
GDB to debug their applications.

I started to think maybe we should make GDB more resistant to
demangler crashes, to keep our users in business.  I attached a basic
patch I've worked on as a proof of concept.  I'm aware this needs
portability work, a ChangeLog, and maybe error message throttling.
More importantly, I'm also aware that none of the wrapped code uses
cleanups, so caught failures will leak memory.  Normally I wouldn't
consider this appropriate, but I think here the benefit may outweigh
the problems.

Example sessions with the attached patch applied, using the symbol
from https://sourceware.org/bugzilla/show_bug.cgi?id=16817:

| (gdb) set lang c++
| (gdb) maint demangle _QueueNotification_QueueController__$4PPPPPPPM_A_INotice___Z
| warning: internal error: demangler failed with signal 11
| Can't demangle "_QueueNotification_QueueController__$4PPPPPPPM_A_INotice___Z"

and:

| (gdb) add-symbol-file /home/gary/a.out 0
| add symbol table from file "/home/gary/a.out" at
|         .text_addr = 0x0
| (y or n) y
| Reading symbols from /home/gary/a.out...warning: internal error: demangler failed with signal 11
| (no debugging symbols found)...done.

Comments please :)

Thanks,
Gary

-- 
http://gbenson.net/

Attachment: demangler-wrap-rfc.patch
Description: Text document