Re: Evolution of ELF symbol management

[Joseph Myers <joseph at codesourcery dot com>]

On Tue, 18 Oct 2016, Florian Weimer wrote:

> I think the above sums up the status quo.  With this message, I want to start
> a discussion why this symbol mangling stops at glibc-internal cross-DSO
> references (or static linking).  Wouldn't other system libraries, such as
> libstdc++, glib, Qt and so on need to do the same thing?  After all, if Qt
> calls foo@GLIBC_2.31, and the main program defines foo (which the static
> linker automatically exports to enable interposition), we almost certainly
> would want Qt to continue to call foo@GLIBC_2.31, and not the potentially
> incompatible implementation of foo in the main program.

We've previously discussed this in the libstdc++ context and I think 
agreed that implementation-namespace versions should be added at least for 
functions used in libstdc++ headers, to allow G++ to stop defining 
_GNU_SOURCE by default 
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=11196> 
<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=51749> 
<https://sourceware.org/ml/libc-alpha/2012-03/msg00311.html>.  I'd 
consider the namespace issues to apply equally to all language runtime 
libraries - so including any symbols used in libstdc++ but not in the 
headers, for example, and in other language runtimes in GCC where there is 
a meaningful question for the relevant languages of certain C symbol names 
being reserved or not reserved.  Language runtimes also include e.g. 
libdfp (on which basis the printf hooks functionality should be exported 
under implementation-namespace names).

Doing it more systematically for glibc function symbols rather than only 
supporting it for particular privileged language runtimes seems reasonable 
to me.

> To keep things simple, I suggest that for all new function symbols, we declare
> __libc_foo in the header file, redirect foo to __libc_foo, export both at the
> same symbol version from the DSO, and make __libc_foo a strong definition and
> foo a weak one.  (We should not add new variable symbols.)

I take it this is __libc_foo independent of which library contains foo (so 
no __libm_foo, __libpthread_foo etc.)?

There are a few existing __libc_foo exports at public symbol versions.  Do 
all those satisfy the rule that where both foo and __libc_foo exist, the 
latest version of foo and the latest version of __libc_foo are aliases or 
otherwise have the same semantics?  (It would seem very confusing for old 
and new __libc_* symbols to follow different rules in that regard.)

What should be done where the symbol is only added in the implementation 
namespace - symbols for use in redirection for different standard 
versions, macros, inline functions or *_nonshared.a, for example?  Should 
future such symbols also use the __libc_foo namespace (unless there are 
ABI reasons for something else, e.g. the libmvec functions) (so if such a 
practice were implemented before glibc 2.25 came out, __iscanonicall would 
change to __libc_iscanonicall, etc.), or continue being __foo?

What about compilers that do not support redirection?  Right now we have 
many individual #defines in the case where __REDIRECT is not supported.  
If we required support for asm redirection in compilers using the glibc 
headers, it would be possible to define a macro to declare both foo and 
__libc_foo, with the same type and the same attributes (and the same throw 
() information for C++), and do the redirection, all with one macro call.  
Otherwise you get a lot of repetitive boilerplate in headers for every 
such function, since a macro cannot generate a #define of another macro.  
Or you say that compilers without redirection support don't get any of 
these redirections, since they are not semantically required.

(When you're dealing with API issues as well as ABI then the macro 
solution runs into complications with wanting to declare __libc_foo 
unconditionally for use in libstdc++ headers, but foo only when the right 
feature test macros are defined.  Those complications can certainly be 
resolved, e.g. with macros __<something>_GNU to do the declaration whose 
definitions depend on the feature test macros defined, and a first 
solution might well only deal with the ABI issues and leave the API ones 
for later.)

Being able to make all the declarations with a single macro is attractive, 
since right now I'm sure that lots of the declarations in internal 
include/ headers are in fact suboptimal because they are missing 
attributes present on the public declarations.  It would also have the 
potential for defining variants of such macros in future that also do 
*_hidden_proto (for public and internal function names) when building 
glibc.  Recall that *_hidden_* are still needed even for internal function 
names, whether or not those names are exported - if exported, failure to 
use *_hidden_* will be visible through localplt test failures, but if not 
exported, less efficient code is still generated in the caller on 32-bit 
x86 if the function isn't visibly hidden 
<https://sourceware.org/bugzilla/show_bug.cgi?id=18822>.

(In turn, that would allow us to move towards the desired direction of 
eliminating most of the include/ header wrappers so tests build in 
something much more like a normal installed glibc environment.  I think 
it's already understood that declarations that aren't just hidden_proto 
(foo) or declaring __foo for a function foo in the public header ought to 
go in an entirely separate header, not one of those wrappers, and 
appropriate macros for function declarations in installed headers could 
allow eliminating the remaining appropriate contents from the wrappers - 
subject to the issue of the declarations there being for old-style names 
for internal functions, not for __libc_*.)

Features that are de facto required for using glibc headers already 
include (non-exhaustive list):

* C89 or C++98.

* long long.

* Flexible array members - including the ability for a struct with a 
flexible array member to be followed by another member in a containing 
struct, which is not a standard C feature.  (See _G_config.h's _G_iconv_t; 
struct __gconv_info has a flexible array member and is followed by another 
member in a struct.  I don't know if use of a compiler that gets the 
fallback array[1] for a flexible array member would result in any ABI 
issues for a user of glibc, or if the ABI in question is purely internal.)

* Some headers require anonymous structs / unions.

* Various macros in various headers may also require other features such 
as __typeof and statement expressions.

In fact we have evidence 
<https://sourceware.org/ml/libc-alpha/2014-09/msg00017.html> that the 
headers have had problems for a long time for compilers not defining 
__GNUC__, and those include problems relating to redirection.

> For existing symbols, we only do this if we receive reports of conflicts
> causing problems in the field.  In this case, we add __libc_foo and the
> redirect to the header file, and use the current symbol version for the
> __libc_foo export (not the one of foo).

"causing problems in the field" should be broadly interpreted there - to 
allow adding lots of such functions if someone identifies what's needed to 
make the libstdc++ headers or libraries namespace-clean, for example, or 
for fixing the namespace issues described in 
<https://sourceware.org/bugzilla/show_bug.cgi?id=14106>.

What should be done in the case where __foo already has an export at a 
public symbol version (and we have a use for __libc_foo)?  Should we 
arrange for __foo to be declared (with associated redirections) and say 
people should be using that, or add __libc_foo as well?  What about where 
__foo is already exported, but that export is a compat symbol (if there 
are any such cases)?  Making it not a compat symbol would run into needing 
new exports at new versions on platforms postdating the version where it 
was made a compat symbol, and you don't want the API to be __foo on some 
platforms and __libc_foo on others.

-- 
Joseph S. Myers
joseph@codesourcery.com