This is the mail archive of the
libc-help@sourceware.org
mailing list for the glibc project.
Re: Help: shutdown(..., SHUT_WR) on TCP sockets
- From: Sergey Organov <sorganov at gmail dot com>
- To: Florian Weimer <fweimer at redhat dot com>
- Cc: libc-help at sourceware dot org
- Date: Fri, 25 Oct 2019 07:52:52 +0300
- Subject: Re: Help: shutdown(..., SHUT_WR) on TCP sockets
- References: <qblfrh$4m4i$1@blaine.gmane.org> <87woippfuu.fsf@oldenburg2.str.redhat.com> <87a7flxlvp.fsf@javad.com> <87ef4xnlnv.fsf@oldenburg2.str.redhat.com> <87woipuld0.fsf@javad.com> <87sgt9i5s5.fsf@oldenburg2.str.redhat.com> <87k1elp36e.fsf@javad.com> <87pnmkr7do.fsf@oldenburg2.str.redhat.com> <87tvbwsj90.fsf@osv.gnss.ru> <87h87woba2.fsf@oldenburg2.str.redhat.com> <8736jgsia7.fsf@osv.gnss.ru>
Hello Florian,
I've sent this to the mailing list only the first time, sorry, --
resend.
Sergey Organov <sorganov@gmail.com> writes:
> Florian Weimer <fweimer@redhat.com> writes:
>
>> * Sergey Organov:
>>
>>> Florian Weimer <fweimer@redhat.com> writes:
>>>
>>>> * Sergey Organov:
>>>>
>>>>> Maybe, but honestly, I fail to see how adding 'shutdown(fd, SHUT_WR)'
>>>>> anywhere before 'close(fd)' can do things worse from the POV of data
>>>>> delivery to the other end.
>>>>>
>>>>> What I observe is that either:
>>>>>
>>>>> sleep(1);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> or:
>>>>>
>>>>> sleep(1);
>>>>> shutdown(fd, SHUT_WR);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> deliver all the data, while:
>>>>>
>>>>> shutdown(fd, SHUT_WR);
>>>>> sleep(1);
>>>>> close(fd);
>>>>> exit(0);
>>>>>
>>>>> cuts some of the data (read() returns 0 on the other end indicating
>>>>> closed socket).
>>>>
>>>> Wait, you get less data with the last sequence? And you do not write to
>>>> fd after the shutdown call?
>>>
>>> Yes, and this is single-threaded application so I'm very sure I don't
>>> write anything myself.
>>>
>>> However, as I already mentioned, ioctl(fd, TIOCOUTQ, &v) gives 0 before
>>> shutdown() and 1 right after shutdown(), that suggests shutdown()
>>> itself writes something. Dunno if it's expected.
>>
>> It needs to queue the FIN segment, for the connection teardown
>> handshake.
>>
>>>>> Another mystery is that 'ioctl(fd, TIOCOUTQ, &v)' gives 0 in all the
>>>>> above cases when put at the beginning of the above sequences[*],
>>>>> indicating that there are no pended data, so there should be nothing to
>>>>> loose in the first place, one way or another.
>>>>
>>>> Yes, I agree that data loss should not happen in these cases.
>>>
>>> It looks like shutdown() does something that is transferred to receiver
>>> and makes it drop some amount of recently received data that receiving
>>> application didn't read yet. I.e., it's probably receiving end that
>>> drops the data?
>>
>> Do you see any RST segments in packet captures? As I said, RST is not
>> expected to be ordered with respect to data delivery, so if an RST
>> segment is generated, then it would explain the data loss.
I've got time to get back to the issue, and what I see is the following.
When shutdown(fd, SHUT_WR) is called without delay on the sending end, the sending
terminates early like this:
[...]
08:14:25.741054 IP (tos 0x0, ttl 64, id 23482, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e17 (correct), seq 540:541, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741331 IP (tos 0x0, ttl 64, id 23483, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e16 (correct), seq 541:542, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.741728 IP (tos 0x0, ttl 64, id 23484, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43076: Flags [P.], cksum 0x5e15 (correct), seq 542:543, ack 16504, win 1215, options [nop,nop,TS val 188980857 ecr 1249457135], length 1
08:14:25.743013 IP (tos 0x0, ttl 64, id 23485, offset 0, flags [DF], proto TCP (6), length 52)
omega-51da5d.gnss.ru.8002 > osv.43076: Flags [F.], cksum 0x641b (correct), seq 543, ack 16504, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0
08:14:25.743644 IP (tos 0x0, ttl 64, id 23486, offset 0, flags [DF], proto TCP (6), length 52)
omega-51da5d.gnss.ru.8002 > osv.43076: Flags [.], cksum 0x6414 (correct), seq 544, ack 16511, win 1215, options [nop,nop,TS val 188980858 ecr 1249457135], length 0
whereas in the "good" case, where shutdown(fd, SHUT_WR) is called after
1-second delay, sending terminates much later, like this:
[...]
08:04:50.446074 IP (tos 0x0, ttl 64, id 34857, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc795 (correct), seq 697:698, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446339 IP (tos 0x0, ttl 64, id 34858, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc794 (correct), seq 698:699, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.446738 IP (tos 0x0, ttl 64, id 34859, offset 0, flags [DF], proto TCP (6), length 53)
omega-51da5d.gnss.ru.8002 > osv.43048: Flags [P.], cksum 0xc793 (correct), seq 699:700, ack 16503, win 1385, options [nop,nop,TS val 188923385 ecr 1249313311], length 1
08:04:50.459717 IP (tos 0x0, ttl 64, id 34860, offset 0, flags [DF], proto TCP (6), length 64)
omega-51da5d.gnss.ru.8002 > osv.43048: Flags [.], cksum 0x0fed (correct), ack 16510, win 1385, options [nop,nop,TS val 188923386 ecr 1249313311,nop,nop,sack 1 {16509:16510}], length 0
08:04:51.384289 IP (tos 0x0, ttl 64, id 34861, offset 0, flags [DF], proto TCP (6), length 52)
omega-51da5d.gnss.ru.8002 > osv.43048: Flags [F.], cksum 0xcd35 (correct), seq 700, ack 16510, win 1385, options [nop,nop,TS val 188923479 ecr 1249313311], length 0
To me it looks like it's in fact the sending end of the connection that
drops a lot of yet unsent packets (packets 543--700) when shutdown() is
called.
What is also surprising is that TIOCOUTQ requests put like this:
int v1 = -1, v2 = -1;
ioctl(fd, TIOCOUTQ, &v1);
shutdown(fd, SHUT_WR);
ioctl(fd, TIOCOUTQ, &v2);
printf("$$OUT %d %d\n", v1, v2);
sleep(1);
close(fd);
exit(0);
result in
$$OUT 3 2
debug output in the "bad" case (that doesn't match the number of dropped
data at all).
Any thoughts? Does it look like a bug to be reported?
Thanks,
-- Sergey