Re: dlopen and memory order guarantees with threads

[edA-qa mort-ora-y <eda-qa at disemia dot com>]

Great, that's very helpful. Essentially the loader relies on mmap to
provide all visibility guarantees across the processors. Do you know if
there is any doc/spec which makes this guarantee explicit (the man page
and POSIX references don't appear to explicitly mention this aspect).

To ensure I tracked this correctly through the code, the actual call to
__mmap is in the "_dl_map_object_from_fd" function in <elf/dl-load.c>.
Is that correct (at least for ELF files)?

Somewhat unrelated, but if I read this correctly then, since the dl-open
just mmaps the file, that means glibc is solely responsible for handling
variables marked as thread-local (that is, not the OS).

On 03/03/2012 05:35 PM, Carlos O'Donell wrote:
> On Sat, Mar 3, 2012 at 1:14 AM, edA-qa mort-ora-y <eda-qa@disemia.com> wrote:
>> I call dlopen, then I call dlsym to get the address of some object, lets
>> say a C-string literal. Is this pointer immediately safe to use in
>> another thread of the same application (with respect to memory
>> ordering/visibility)?
> 
> Yes. Though this is guaranteed by the kernel. The file,
> and thus the contents of the C-string will have been
> mmap'd into the process and thus all threads.
> 
>> My assumption is yes, but I don't see this as an explicit guarantee. If
>> I were to allocate memory on my own, and write a string to it, I would
>> need to have both threads issue a synchronization command (atomic memory
>> order, mutex, fence, etc.) in order for the other thread to safely use
>> the pointer to the string.*
> 
> The dynamic loader functions are thread-safe.
> 
> The call to dlsym can't be made until dlopen returns
> with a handle for the caller to use.
> 
> At that time the file has been mapped into the process'
> address space and the kernel ensures all threads see
> the same mapping.
> 
> Your question about writing to shared memory is a
> different issue. If dlopen/dlsym were actually
> implemented as an fread to memory, yes there would have
> to be barries in place to ensure that after dlopen all
> threads see the same memory.
> 
> However in glibc dlopen is not implemented as an fread
> from disk.
> 
>> How does dlopen, assuming it does, guarantee that any pointer returned
>> from dlsym points only to memory which is completely visible with
>> current data from any thread running on any processor?
> 
> It doesn't. The kernel does. This is part of the memory
> mapping API in the kernel. It ensures that mapped memory
> is consistent for all threads. It's not an easy guarantee
> to make.
> 
>> Is this synchronization provided by glibc or by the underling OS (in
>> which case I'd be happy with a Linux explanation here). I tried
>> following the glibc source code, but I don't quite understand what
>> happens yet (what does GLRO(dl_open) call?)
> 
> Yes, the synchronization is provided by the underlying
> OS, Linux in this case, or the Hurd or FreeBSD.
> 
> This is not a trivial problem to solve, and you must
> understand the OSs cache/tlb flush mechanisms and
> how they are used to provide coherency.
> 
> To answer your other question...
> 
> The GLRO() macro is used to access global or local
> read-only data, see sysdeps/generic/ldsodefs.h.
> 
> The macro GLRO(dl_open) returns a function pointer
> to dl_open and the call is made through that function
> pointer. For a treatise on function pointers I point
> you at the ISO C standard or Google.
> 
>> (*Let's assume an architecture were memory ordering/visibility could
>> actually cause a problem, since on x86 we should be fine with my example
>> even without syncing. Of course also assume multiple processors with the
>> threads truly running concurrently.)
> 
> I recommend you start by reading through and trying
> to understand as much as you can about cache and
> TLB architectures on modern processors and what
> problems they try to solve.
> 
> Generally the hardware also solves the more complicated
> problem of cache-coherency across multiple processors.
> 
> Cheers,
> Carlos.
> 
> 


-- 
edA-qa mort-ora-y
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
Sign: Please digitally sign your emails.
Encrypt: I'm also happy to receive encrypted mail.

Attachment: signature.asc
Description: OpenPGP digital signature