Re: Problem with atexit and _dl_fini

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

On 22/05/2019 07:22, Nat! wrote:
> Adhemerval Zanella schrieb:
> 
>>
>> Could you provide a testcase that stress the issue you are seeing?  At
>> least on glibc it does have a testcase that does pretty much what you
>> described, dlfcn/bug-atexit1-lib.c and dlfcn/bug-atexit1.c.
> 
> I tried to simplify it, but I failed and gave up (see below).
> 
>>
>> It created a shared library with an exported symbol that registers at
>> lot of atexit function and the main program dlopen and dclose it and
>> checks if the atexit handlers are indeed called in the correct order.
>> It does not use any C++, so there is no __cxa_finalize involved.
> 
> The use of __cxa_finalize is something I observed in my stacktrace
> (see previous mails).
> 
>>
>> Also, on your debug information I must confess it is confusing what
>> exactly you are expecting and what exactly your program is doing.
>> For instance I am not understanding the part 'I registered my two
>> function with `__internal_atexit`'. Are you trying to calling it
>> directly? Or are you calling __cxa_atexit? Keep in mind that
>> __cxa_atexit calls are generated by compiler itself to destruct
>> global objects.
>>
> 
> I am really just using `atexit`. I was trying to explain, what the internal glib problem is, that I observed setting breakpoints and stepping through the code.
> 
> What conceptually I am doing is to install on-demand `atexit` handlers during the load of shared libraries. These are then used for debugging.
> 
> Basically like this (typed by hand):
> 
> liba.so:
> 
> void   a( void)
> {
> }
> 
> __attribute__((constructor))
> static void   check( void)
> {
>    if( getenv( "DO_B_ON_EXIT"))
>       atexit( a);
> }
> 
> 
> libb.so:
> 
> void   b( void)
> {
> }
> 
> __attribute__((constructor))
> static void   check( void)
> {
>    if( getenv( "DO_A_ON_EXIT"))
>       atexit( b);
> }
> 
> I tried this with a simple setup and that doesn't create problems as such. But dl-fini re-sorts the dependencies sometimes, and then
> the atexit order is compromised and that is what I am running into.
> 

Are you sure that you calling the atexit in the order you expect? Because by
explicit linking it will depend on the order of the shared object you are
passing on linking command.

Using the example (I just instrumented both 'a' and 'b' to prints the function
name using write on STDERR_FILENO):

$ gcc -Wall test.c -o test -L`pwd` -Wl,--no-as-needed -lliba -llibb
$ LD_LIBRARY_PATH=. DO_A_ON_EXIT=1 DO_B_ON_EXIT=1 ./test 
a
b

$ gcc -Wall test.c -o test -L`pwd` -Wl,--no-as-needed -llibb -lliba
$ LD_LIBRARY_PATH=. DO_A_ON_EXIT=1 DO_B_ON_EXIT=1 ./test 
b
a

Is is also similar to a dlopen case:

--
#include <dlfcn.h>
#include <assert.h>

int main (void)
{ 
  void *handle_a = dlopen ("libliba.so", RTLD_NOW);
  assert (handle_a);
  void *handle_b = dlopen ("liblibb.so", RTLD_NOW);
  assert (handle_b);
}
--

$ gcc -Wall test-dlopen.c -o test-dlopen -ldl
$ LD_LIBRARY_PATH=. DO_A_ON_EXIT=1 DO_B_ON_EXIT=1 ./test-dlopen 
b
a

What you might seeing is maybe a implicit dependency that make loading 
in a specific order even if you try to explicit issue them on linking 
command.  Using the starting example, if you link liblibb.so with 
libliba.so as a dependency:

$ gcc -Wall -shared -fpic liba.c -o libliba.so
$ gcc -Wall -shared -fpic libb.c -o liblibb.so -L`pwd` -Wl,--no-as-needed -lliba

It does not matter how you link the resulting program, the atexit handler will
be registered in the same order:

$ gcc -Wall test.c -o test -L`pwd` -Wl,--no-as-needed -llibb -lliba
$ LD_LIBRARY_PATH=. DO_A_ON_EXIT=1 DO_B_ON_EXIT=1 ./test
b
a
$ gcc -Wall test.c -o test -L`pwd` -Wl,--no-as-needed -lliba -llibb
$ LD_LIBRARY_PATH=. DO_A_ON_EXIT=1 DO_B_ON_EXIT=1 ./test
b
a

I really think it is something related to your build because glibc does not
actually reorder the internal __exit_funcs list.  AFAIK once it is inserted at
stdlib/on_exit.c by __internal_atexit it is deregistered at stdlib/cxa_finalize.c.

So if you could actually show the issue your are observing, please open a bug
report.  Otherwise I am not seeing an actually issue with atexit handlers
here.