[RFC] Global breakpoints

[Stan Shebs <stan at codesourcery dot com>]

Our next planned addition to multi-process debugging in GDB is the ability
to have "global breakpoints".  Inspired by a similar feature in an unnamed
RTOS, a global breakpoint applies to multiple processes, possibly
including future processes.

The utility in a parallel-processing or multi-core environment should
be obvious; if we have, say, 64 processes running already, and start up
GDB as it is today, we can't catch the first use of function
grab_resource without individually attaching to and setting the same
breakpoint in every one of those processes.  With a global breakpoint,
we can instead say something like

  break grab_resource process 2150-2300

and then just sit back and wait for the first hit, at which point GDB
attaches to the hitting process, reports the stop, and we proceed with
our debugging session in the usual way.

(Implementors will immediately see a dozen difficulties making this
work in GNU/Linux - more on that below.)

Beyond the basic enumeration of processes by pid, we also want to
support asking for a global breakpoint in all existing processes
and/or all future processes.  On top of that, we want to be able to
ask for a location in a specific shared library or executable, which
means adding to location syntax.  Also, with sufficient privileges, we
want to be able to set breakpoints in any user's process. (No, this is
not just for sysadmin entertainment, consider embedded Linux, and in
fact part of the project is to augment GDBserver.)

There are two major additions to user interface.  The first is the
addition of a "process" suffix to the break command.  Although the old
HPD standard calls for multi-process commands to be prefixed with a
square-bracketed "process/thread set specification", the suffix fits
better with our break command, which already uses "thread" and "if" in
that position.  The actual process specification is a combination of
comma-separated pids, pid ranges, and symbolic names - at least "all"
for all current processes, and "future" for new processes.

The second interface addition is an optional exec/shlib specification
that is a prefix to file:line-type locations.  The desirable syntax is
not obvious to me; HPD specifies '#' as delimiters for location parts,
but as it is the comment char in GDB, there is some ambiguity.  Since
locations don't appear in general expressions, we do have a variety of
options.

There will also be MI interface to specify for all this.

Implementation in Linux is going to be kind of freaky.  It's not
desirable to have GDB attach to every process in sight preemptively,
nor to have GDBserver do it, which means that we're talking about
something like a mini GDB stub embedded in a kernel module.  It has to
detect process creation and shared library loading, insert traps,
catch trap hits, and notify GDB so that it can attach and convert the
trap hit into a breakpoint stop.  It will also need to check on
permissions, and be smart about handling traps inserted in
GDB/GDBserver or any of the libraries they call; I think it will work
to *report* that the global breakpoint was hit in the debugger's
process first, it just can't stop anything.

The kernel module gets really messy when you consider the possibility
of several people setting global breakpoints at the same time, possibly
even in the same processes!

Another possible strategy is to hook into the dynamic loader, and it
has the advantage of (probably) simplifying breakpoint insertion for
library code, although it's not as clear to me how it gets GDB's
attention.

Since we're just now starting on this work, any commentary would be
much appreciated, particularly in relation to how to make this work
reasonably without excessive kernel hacking.

Stan