Re: Principles for API sources

["Carlos O'Donell" <carlos at redhat dot com>]

On 11/06/2015 09:07 AM, Joseph Myers wrote:
> Various recent proposals for new APIs taken from existing sources of APIs 
> seem to have got stuck for similar reasons.  This includes Linux kernel 
> syscall wrappers as well as explicit_bzero and strlcpy.  I'd like to 
> propose some general principles for such APIs in the hopes of unsticking 
> those discussions - if we can get consensus on principles (consensus does 
> not mean unanimity; consensus does not mean waiting indefinitely for 
> people who might want to contribute to the discussion at some 
> indeterminate point in the future) then we don't need to redebate the 
> principles for each proposed API, just consider how they apply to that 
> API.
> 
> All of the following is only about APIs, for the OS-independent GNU API (I 
> think the idea of libinux-syscalls.so.1 for OS-specific Linux syscall 
> wrappers was uncontroversial).  It does not address what library an API 
> goes in, or other ABI issues, although in most cases the choice of library 
> will be obvious.  And all the usual patch requirements apply to any API 
> addition.
> 
> * Required ISO C and POSIX APIs (in current versions, and in previous 
> versions with very rare exceptions where there is particular difficulty in 
> supporting an old API) are all assumed appropriate for inclusion in glibc 
> (I think this is uncontroversial).

Agreed.
 
> * Optional ISO C and POSIX APIs, and APIs from ISO C TRs / TSs / standards 
> outside the main C standard, require a bit more consideration, but that 
> should generally be at the level of considering a complete set of APIs 
> together, and with a starting point in favour of inclusion if the nature 
> of the APIs is very closely tied to existing glibc APIs or aspects of them 
> make them hard to implement correctly independently of libc.  (So we might 
> discuss the desirability of C11 Annex K support in glibc, for example, and 
> whether such APIs should generally go in a separate libc_s shared library 
> built by glibc, but should not get into piecemeal debate about individual 
> APIs there.)

Agreed.

> * Historical practice was that glibc adopted both BSD and SVID APIs, 
> including multiple APIs for the same thing, and until recently adopted 
> APIs for most Linux syscalls (although some of those went in libc.so for 
> GNU/Linux only).  My claim is that we have now swung much too far the 
> other way in conservatism about adopting existing APIs, and that this is 
> not beneficial to glibc or glibc users.

Agreed.

Given the caveat that some specific syscalls are simply not useful and
or conflict with existing APIs and the best example I have is the often
requested gettid() which I think *could* exist, but we need an API design
review to harmonize this with existing pid_t based interfaces before
letting users use it and get confused.

> * Specifically, I think the following should be considered appropriate 
> sources for APIs to be added to the OS-independent GNU API in glibc, where 
> the APIs in question are the general sort of functionality traditionally 
> included in libc and associated libraries on Unix-like systems: Linux 
> kernel syscalls, where the functionality would plausibly be meaningful 
> under other kernels; modern BSD systems; other Unix-like operating 
> systems.  This is in order of preference: APIs from Linux syscalls have a 
> stronger expectation of appropriateness than those from modern BSD 
> systems, which have a stronger expectation of appropriateness than those 
> from other Unix-like operating systems.

Agreed.

> * The fact that an API already exists and is being used in free software 
> (in a context where it isn't deprecated) should be considered a point in 
> favour of inclusion of that API, that can counterbalance arguments about 
> imperfections in the API design; the more widespread the usage is, the 
> more important it is as an argument for inclusion.  Deficiencies in API 
> design, or issues of consistency with other APIs in glibc, can still be 
> relevant issues.  But whether they outweigh usage depends on how serious 
> the deficiencies are, and how widespread the usage is.

Agreed.

> * Where there are choices of multiple APIs in an area, usage information 
> (such as from Zack's analysis for explicit_bzero) is valuable for choosing 
> between them.  (Where multiple APIs are in use there may sometimes be a 
> case for supporting more than one of them, but that needs a stronger case 
> than for adding one API in an area from an existing API source.)

Agreed.

> * The following are all points in favour of inclusion of an API because 
> they would make a good external implementation more difficult: (a) natural 
> integration with fortification support; (b) natural integration with 
> thread cancellation; (c) implementation involves low-level information 
> about syscall calling conventions; (d) any other way in which the function 
> is naturally integrated with libc infrastructure rather than being 
> definable completely independently.

Agreed.

Syscall cancellation is the strongest reason for syscall wrapper integration
into glibc (either directly or in another related shared library).

> * That an API is not fully general - that there is some problem in the 
> area of the API that it does not fully solve - is only a weak point 
> against that API.  This is especially the case if the API (notwithstanding 
> its limitations) is already widely used.  This is also especially the case 
> for APIs related to security hardening (making existing code more robust 
> against exploitation in the presence of bugs, through local changes), 
> where in general absolute protection does not exist.

Agreed.

> * We should respect that other communities do have good processes for API 
> review, in particular as regards Linux kernel syscalls added in the past 
> few years, so that most of the design review for a new API has already 
> been done if that API comes to us from an external source.

Agreed.

That doesn't preclude doing our own secondary review of the API, but we
need to be directly involved in linux-api and other lists.

For example, I'm currently engaging Intel on the new Memory Protection
Keys to try get a coherent API that all applications could use, and
because it's a shared resource it makes sense for the APIs to be in glibc.
 
> * On the above basis I would accept most non-deprecated Linux syscalls, 
> except where particularly OS-specific, while much more selectively 
> accepting widely-used BSD APIs, especially in the security hardening area 
> (including explicit_bzero, strlcpy and strlcat, for which I think the 
> widespread use, and greater suitability for hardening existing code than 
> redesigning its memory allocation strategies, outweighs the imperfections 
> of the APIs).
 
Agreed, with the exception for tid/thread since I think that has some
special consideration given POSIX Threads interactions. Similarly I think
we need to be careful about scheduling APIs and their overlap with POSIX.

Cheers,
Carlos.