Re: Help: looking for equivalent syscall to shutdown() but for unix pipes

[Godmar Back <godmar at gmail dot com>]

On Fri, Jan 18, 2019 at 5:11 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Alexandre Bique:
>
> > On Thu, Jan 17, 2019 at 9:48 AM Florian Weimer <fweimer@redhat.com> wrote:
> >>
> >>
> >> You need some sort of framing layer, so that you can transmit both data
> >> and an EOF indicator.
> >
> > Thank you very much.
> >
> > I'm not very enthusiast with the idea of sending an EOF message,
> > because the original producer comes from an other process, so there
> > could be all sorts of corrupted data over the pipe, unless maybe it is
> > using packets instead of streams. But in our case it is using streams.
> >
> > I was thinking that I could use poll on the file descriptor and then
> > just check a flag before reading the pipe. Now to wake up the poll()
> > syscall I would need an eventfd which is an additional opened file,
> > not awesome.
>
> How do you detect that you are at the end of the stream if you do not
> have an EOF indicator?  If the other process crashes for any reason, you
> are left with truncated data if you don't have an EOF indicator.
>
> In the past, I dealt with the crash problem by launching the producer
> process from the consumer, and commit to processing the data in the
> consumer only after the producer has exited with zero status.  Maybe
> that's an approach you could use?
>

I agree with Florian that you need to think about the higher-level
semantics and how you can reconcile the notion of a "stream" with
absolutely no message boundaries with needing to know when the data
has been sent completely.

Pretty much any protocol that's layered on a byte stream transport has
this: HTTP, for instance, creates message boundaries via a message
length sent in the header.

Waiting for the consumer to exit with zero of course raises potential
buffering issues.  You can't wait to read until the consumer exits (or
else you could deadlock), so you have to buffer, in which case you
need to think about how much. But that's perhaps a problem you're
facing anyway.

 - Godmar