Re: Principles for API sources

[Rich Felker <dalias at libc dot org>]

On Fri, Nov 06, 2015 at 02:07:20PM +0000, Joseph Myers wrote:
> Various recent proposals for new APIs taken from existing sources of APIs 
> seem to have got stuck for similar reasons.  This includes Linux kernel 
> syscall wrappers as well as explicit_bzero and strlcpy.  I'd like to 
> propose some general principles for such APIs in the hopes of unsticking 
> those discussions - if we can get consensus on principles (consensus does 
> not mean unanimity; consensus does not mean waiting indefinitely for 
> people who might want to contribute to the discussion at some 
> indeterminate point in the future) then we don't need to redebate the 
> principles for each proposed API, just consider how they apply to that 
> API.
> 
> All of the following is only about APIs, for the OS-independent GNU API (I 
> think the idea of libinux-syscalls.so.1 for OS-specific Linux syscall 
> wrappers was uncontroversial).  It does not address what library an API 
> goes in, or other ABI issues, although in most cases the choice of library 
> will be obvious.  And all the usual patch requirements apply to any API 
> addition.

I generally agree with what you've written, so I'm just replying to
the parts I disagree with or have specific additional comments on. See
below:

> * Optional ISO C and POSIX APIs, and APIs from ISO C TRs / TSs / standards 
> outside the main C standard, require a bit more consideration, but that 
> should generally be at the level of considering a complete set of APIs 
> together, and with a starting point in favour of inclusion if the nature 
> of the APIs is very closely tied to existing glibc APIs or aspects of them 
> make them hard to implement correctly independently of libc.  (So we might 
> discuss the desirability of C11 Annex K support in glibc, for example, and 
> whether such APIs should generally go in a separate libc_s shared library 
> built by glibc, but should not get into piecemeal debate about individual 
> APIs there.)

I think there may be cases where it makes sense to adopt some
functions but not others from an option group, when the option group
was poorly categorized. The most obvious example is having ioctl but
not STREAMS. ;-) At one point I thought treating Annex K this way
might make sense, but there seems to be widespread consensus outside
glibc that Annex K is not the right way forward (bad interactions with
runtime constraint handlers) and that, if similar functionality is
needed but missing from POSIX, it should be proposed for addition as
separate functions not tied to Annex K.

> * Where there are choices of multiple APIs in an area, usage information 
> (such as from Zack's analysis for explicit_bzero) is valuable for choosing 
> between them.  (Where multiple APIs are in use there may sometimes be a 
> case for supporting more than one of them, but that needs a stronger case 
> than for adding one API in an area from an existing API source.)

One important specialization of this issue that's come up with musl is
when there are multiple historic APIs with the same name, but
different signatures. For musl's purposes, this is a strong cause of
exclusion of the interface, at least unless/until other implementors
begin to build consensus on which one to deprecate (see the qsort_r
issue for example). glibc should probably try to adopt some consensus
position on this issue.

> * We should respect that other communities do have good processes for API 
> review, in particular as regards Linux kernel syscalls added in the past 
> few years, so that most of the design review for a new API has already 
> been done if that API comes to us from an external source.

I disagree that this is the case; I think we should be actively
pushing for a cross-implementation consensus based process to improve
the situation here. In particular Linux has introduced a number of
very buggy APIs due to subtle type errors that would have been caught
by any proper review: things like sendmmsg/recvmmsg, x32 timespec,
etc. OpenBSD is also very quick to add new APIs as a quick fix
(usually for the latest security concern) without good design. But
rather than making scapegoats of them, I think we should be trying to
establish the importance of portable APIs and reaching out to other
implementors with the attitude that we want to adopt good APIs they
add, but that we also want to avoid getting locked in to design
mistakes. As for the strl* issue, which is a good example: yes, these
could have been done a lot better, but they're widely used now and it
does not make sense to keep shunning them as a political/ideological
statement about OpenBSD's design of them. But I think other
implementors would be more interested in spending the effort to design
stuff well if they knew that good design would make it easier to get
glibc and other implementations to adopt their ideas.

Rich