Re: lazy library loading

[Carlos O'Donell <carlos at systemhalted dot org>]

On 02/19/2013 11:06 AM, Paul Pluzhnikov wrote:
> On Tue, Feb 19, 2013 at 7:47 AM, Carlos O'Donell
> <carlos@systemhalted.org> wrote:
> 
>> (b) Enhance the resolver such that if the symbol requested is in a library
>>     that hasn't been loaded that the library should be loaded before
>> trying to resolve the symbol.
> 
> I've read the referenced Oracle page, and I still don't understand how
> lazy loading works.
> 
> Here at Google we too have executables (unit tests) with 5000+ shared libraries
> 
> Such executables start *very* slowly with default build of glibc, and
> we carry a local patch that drastically speeds it up. Earlier form of
> that patch has been proposed and rejected here:
> http://www.sourceware.org/ml/libc-alpha/2006-01/msg00018.html

Just to be clear, no code was posted, and nothing was rejected,
it was simply stated that: optimizations for corner cases should
not penalize the common case. It might be possible to do what you
want without penalizing the common case, but without posted code,
and without review, we have nothing to talk about.

The community would be very happy to have Google contribute 
enhancements to the dynamic loader that allow it to run faster
in general. I would be more than happy to help review such patches.

> The problem is that when a call to e.g. malloc is made from anywhere,
> then the *entire* list of shared libraries is *linearly* scanned for
> the definition of that symbol, and this process repeats for every
> symbol that (eventually) comes from libc.so.6 at the end of the link
> chain.

What is the problem? The linear scan? Yes, it's slow.

> For the delayed loading to be effective, you would somehow have to
> know that the delay-loaded library does *not* provide the symbol you
> are looking for.

Correct.
 
> That implies that you'd have to record "library:symbol" unresolved
> references instead of just "symbol" (similar to what AIX and Win32 do;
> changes ELF linking model, requires changes to binutils), or you'd
> have to partially load delay-loaded libraries (so you know what
> symbols they do define), and that would significantly complicate the
> loader.

That's exactly what Andrew proposes in his second bullet point.

> I am not saying that this can't be done, but I think it's
> significantly more complicated project than you make it appear.

Nobody is saying it's easy :-)

Cheers,
Carlos.