Re: [PATCH] Prelimit number of bytes to read in "vFile:pread:"

[Pedro Alves <palves at redhat dot com>]

On 08/19/2015 10:12 PM, Sandra Loosemore wrote:

>>> Unfortunately, I still can't see that ^C is doing anything useful
>>> for me.  It is not coming back to a gdb prompt any sooner and "info
>>> sharedlibrary" afterwards prints the same thing whether or not I've
>>> tried to interrupt it.  This was with unpatched FSF trunk.  How am I
>>> supposed to tell whether ^C did anything?  How are you guys telling
>>> that it is doing something useful?  Is there supposed to be some
>>> sort of message?  If the file transfer from the target is aborted, I
>>> think it should say that.

For me, it just says the usual "Quit":

$ gdb -ex "tar rem :9999"
...
Remote debugging using :9999
^CQuit
(gdb)

That interrupts the main executable transfer.  But from below, I
see we weren't testing the same exact thing.

> Well, here is a clue.  I tried logging the RSP traffic so I could 
> compare the interrupted and non-interrupted behavior.  Aside from 
> differences in PID numbers (etc), the *only* difference is that the log 
> from the interrupted run shows that it's sending a \x03 character to the 
> remote target after it does a vCont, after it has already read the 
> entire contents of libc.so.  Here's the relevant snippet from the 
> transcript:

...

> w $QPassSignals:#f3
> r $OK#9a
> w $vCont;c:p33e.33e#16\x03  <=== here
> r $T05swbreak:;1b:f8f6fe7f;20:d0b1aa2a;thread:p33e.33e;core:0;#89

OK, so ctrl-c while the program is running just causes the target
to report a SIGINT stop.  If you ctrl-c at that time, then it should
make no difference whether GDB is retrieving remote files or not.

The trouble is ctrl-c after the target has stopped for the shared
library load event, and gdb is busy parsing the shared library
(elf headers, debug info, etc.), which involves reading chunks of
the file out of the target side with the vFile:pread packets.

> 
> FAOD, this is a trunk checkout with the patch described above applied 
> and nothing else.  And, the exact sequence of commands I'm using to try 
> to reproduce this is
> 
> <target>-gdb a.out
> target remote ...
> break main
> c
> ^C
> 

Thanks.  Given the recipe, it's easy to reproduce.  Gary, can you
check whether you also see this?

(this is again remote debugging gcc76 on the gcc compile farm
from my office machine)

$ gdb -ex "set verbose on" /tmp/gdbserver -ex "tar rem :9999"
GNU gdb (GDB) 7.10.50.20150820-cvs
Reading symbols from /tmp/gdbserver...done.
Remote debugging using :9999
Created trace state variable $trace_timestamp for target's variable 1.
Reading symbols from target:/lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Reading symbols from system-supplied DSO at 0x7ffff7ff8000...(no debugging symbols found)...done.
0x00007ffff7ddd190 in ?? () from target:/lib64/ld-linux-x86-64.so.2
(gdb) b main
Breakpoint 1 at 0x41200c: file ../../../src/gdb/gdbserver/server.c, line 3635.
(gdb) c
Continuing.
^CReading symbols from target:/lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libc.so.6...(no debugging symbols found)...done.

Breakpoint 1, main (argc=1, argv=0x7fffffffe3d8) at ../../../src/gdb/gdbserver/server.c:3635
3635    ../../../src/gdb/gdbserver/server.c: No such file or directory.
(gdb)

There's a "^C" right after "Continuing.", but it was ignored.


Trying again, but this time debugging gdb, I get:

$ gdb --args $g -ex "set verbose on" /tmp/gdbserver  -ex "tar rem :9999" -ex "b main"
...
Breakpoint 1 at 0x41200c: file ../../../src/gdb/gdbserver/server.c, line 3635.
(gdb) c
Continuing.
Reading symbols from target:/lib/x86_64-linux-gnu/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from target:/lib/x86_64-linux-gnu/libc.so.6...^C
Program received signal SIGINT, Interrupt.
0x0000003615eec5d3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)

stepping into the signal handler is a good way to find what
handler is installed:

(top-gdb) handle SIGINT pass
SIGINT is used by the debugger.
Are you sure you want to change it? (y or n) y
Signal        Stop      Print   Pass to program Description
SIGINT        Yes       Yes     Yes             Interrupt
(top-gdb) s
async_handle_remote_sigint (sig=0) at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:5202
5202    {

That is going to send a ctrl-c request to the target.  Let's resume.

(top-gdb) c
Continuing.
(no debugging symbols found)...done.

Breakpoint 1, main (argc=1, argv=0x7fffffffe3d8) at ../../../src/gdb/gdbserver/server.c:3635
3635    ../../../src/gdb/gdbserver/server.c: No such file or directory.
(gdb)

Again, now with getting a backtrace on that SIGINT:

(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x0000003615eec5d3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:81
81      T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
Missing separate debuginfos, use: debuginfo-install python-libs-2.7.5-16.fc20.x86_64
(top-gdb) bt
#0  0x0000003615eec5d3 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000000057849c in gdb_select (n=10, readfds=0x7fffffffc310, writefds=0x0, exceptfds=0x7fffffffc390, timeout=0x7fffffffc410)
    at /home/pedro/gdb/mygit/build/../src/gdb/posix-hdep.c:31
#2  0x00000000004b685a in ser_base_wait_for (scb=0x191e2f0, timeout=1) at /home/pedro/gdb/mygit/build/../src/gdb/ser-base.c:224
#3  0x00000000004b6af2 in do_ser_base_readchar (scb=0x191e2f0, timeout=10000) at /home/pedro/gdb/mygit/build/../src/gdb/ser-base.c:335
#4  0x00000000004b6c36 in generic_readchar (scb=0x191e2f0, timeout=10000, do_readchar=0x4b6a9d <do_ser_base_readchar>)
    at /home/pedro/gdb/mygit/build/../src/gdb/ser-base.c:410
#5  0x00000000004b6cb0 in ser_base_readchar (scb=0x191e2f0, timeout=10000) at /home/pedro/gdb/mygit/build/../src/gdb/ser-base.c:437
#6  0x000000000075d5dd in serial_readchar (scb=0x191e2f0, timeout=10000) at /home/pedro/gdb/mygit/build/../src/gdb/serial.c:382
#7  0x00000000004e2174 in readchar (timeout=10000) at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:7593
#8  0x00000000004e2cdd in getpkt_or_notif_sane_1 (buf=0xec2650, sizeof_buf=0xec2658, forever=0, expecting_notif=0, is_notif=0x0)
    at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:8134
#9  0x00000000004e2fa1 in getpkt_sane (buf=0xec2650, sizeof_buf=0xec2658, forever=0) at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:8234
#10 0x00000000004e71ad in remote_hostio_send_command (command_bytes=20, which_packet=12, remote_errno=0x7fffffffc7a4, attachment=0x7fffffffc680,
    attachment_len=0x7fffffffc678) at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:10319
#11 0x00000000004e77d0 in remote_hostio_pread_vFile (self=0xdcf120 <remote_ops>, fd=5, read_buf=0x19ed770 "\024", len=16383, offset=0, remote_errno=0x7fffffffc7a4)
    at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:10508
#12 0x00000000004e7a76 in remote_hostio_pread (self=0xdcf120 <remote_ops>, fd=5, read_buf=0x7fffffffc940 "", len=64, offset=0, remote_errno=0x7fffffffc7a4)
    at /home/pedro/gdb/mygit/build/../src/gdb/remote.c:10581
#13 0x0000000000691b98 in target_fileio_pread (fd=0, read_buf=0x7fffffffc940 "", len=64, offset=0, target_errno=0x7fffffffc7a4)
    at /home/pedro/gdb/mygit/build/../src/gdb/target.c:2853
#14 0x0000000000678c68 in gdb_bfd_iovec_fileio_pread (abfd=0x1a63240, stream=0x19203a0, buf=0x7fffffffc940, nbytes=64, offset=0)
    at /home/pedro/gdb/mygit/build/../src/gdb/gdb_bfd.c:294
#15 0x00000000007f7c14 in opncls_bread (abfd=0x1a63240, buf=0x7fffffffc940, nbytes=64) at /home/pedro/gdb/mygit/build/../src/bfd/opncls.c:503
#16 0x00000000007f21a8 in bfd_bread (ptr=0x7fffffffc940, size=64, abfd=0x1a63240) at /home/pedro/gdb/mygit/build/../src/bfd/bfdio.c:196
#17 0x000000000080fe2f in bfd_elf64_object_p (abfd=0x1a63240) at /home/pedro/gdb/mygit/build/../src/bfd/elfcode.h:503
#18 0x00000000007f4acb in bfd_check_format_matches (abfd=0x1a63240, format=bfd_object, matching=0x0) at /home/pedro/gdb/mygit/build/../src/bfd/format.c:305
#19 0x00000000007f4545 in bfd_check_format (abfd=0x1a63240, format=bfd_object) at /home/pedro/gdb/mygit/build/../src/bfd/format.c:94
#20 0x00000000007872f2 in solib_bfd_open (pathname=0x1a5dac0 "/lib/x86_64-linux-gnu/libdl.so.2") at /home/pedro/gdb/mygit/build/../src/gdb/solib.c:504
#21 0x0000000000787418 in solib_map_sections (so=0x1a62ba0) at /home/pedro/gdb/mygit/build/../src/gdb/solib.c:544
#22 0x0000000000787c8f in update_solib_list (from_tty=0, target=0xded2a0 <current_target>) at /home/pedro/gdb/mygit/build/../src/gdb/solib.c:892
#23 0x0000000000787e7c in solib_add (pattern=0x0, from_tty=0, target=0xded2a0 <current_target>, readsyms=1) at /home/pedro/gdb/mygit/build/../src/gdb/solib.c:991
#24 0x00000000007886ba in handle_solib_event () at /home/pedro/gdb/mygit/build/../src/gdb/solib.c:1340
#25 0x00000000005b0c19 in bpstat_stop_status (aspace=0x1095140, bp_addr=140737351955008, ptid=..., ws=0x7fffffffd110)
    at /home/pedro/gdb/mygit/build/../src/gdb/breakpoint.c:5612
#26 0x0000000000638379 in handle_signal_stop (ecs=0x7fffffffd0f0) at /home/pedro/gdb/mygit/build/../src/gdb/infrun.c:5599
#27 0x0000000000637167 in handle_inferior_event_1 (ecs=0x7fffffffd0f0) at /home/pedro/gdb/mygit/build/../src/gdb/infrun.c:5028
#28 0x0000000000637209 in handle_inferior_event (ecs=0x7fffffffd0f0) at /home/pedro/gdb/mygit/build/../src/gdb/infrun.c:5053
...


That is, the target stopped for the shared library event, and GDB is busy
reading the shared libraries.

While this is ongoing, the inferior still owns terminal _input_
(including ctrl-c):

(top-gdb) p terminal_state
$1 = terminal_is_ours_for_output


The "step into signal handler" above shows that at the time the problematic
ctrl-c is pressed, the sigint handler that is registered is
async_handle_remote_sigint.  That's the function that sends the remote-interrupt
packet (\x03) to the remote side.  But the target won't be able to report
a SIGINT until the next time it is resumed (and the kernel reports a SIGINT
to ptrace).  So supposedly, the ctrl-c should be ignored until the
whole file is transferred and gdb re-resumes the target.

I don't see a quick, easy and good way around waiting for the whole file to read.
Having GDB handle ctrl-c differently while it is handling internal events
is already a source of trouble, and the fix always seemed to me to be to make ctrl-c
work like what is happening here.  Specifically, I mean that e.g., say that the
user sets a conditional breakpoint that is evaluated on gdb's side, and then the
target stops for that conditional breakpoint, gdb evaluates the expression,
and then resumes the target again, on and on.  If the user presses ctrl-c just
while the conditional breakpoint's expression is being evaluated, and something along
that code path called target_terminal_ours (thus pulling input/ctrl-c to the
regular "Quit" SIGINT handler), gdb behaves differently (shows a "Quit" and the
debug session ends up likely broken) from when the ctrl-c is pressed while the target
is really running.  I'd argue that the ctrl-c in both cases should be passed
down all the way to the target the same way, and that any internal stop and
breakpoint condition evaluation is magic that should be hidden.  Just like
what is happening here with file reading.

Though having said that, it does look like even that isn't working properly,
as I'd expect this:

(top-gdb) c
Continuing.
(no debugging symbols found)...done.

Breakpoint 1, main (argc=1, argv=0x7fffffffe3d8) at ../../../src/gdb/gdbserver/server.c:3635
3635    ../../../src/gdb/gdbserver/server.c: No such file or directory.
(gdb)

to be slow (that is, the file reading isn't really interrupted), but then
the target stops with SIGINT as soon as gdb resumes it again after reading
the DSOs.  But it is reaching main anyway, and there's no sign
of "Program stopped with SIGINT"...

Not sure where the bug is.  It may be on the gdbserver side.

Thanks,
Pedro Alves