Re: malloc->backtrace->dlopen->malloc deadlock

["Carlos O'Donell" <carlos at redhat dot com>]

On 01/12/2016 04:01 PM, Roland McGrath wrote:
>> Based on the backtrace in the bz, this looks like a different (and
>> new) problem from #16159.  The deadlock seems to be happening on the
>> mtrace-internal LOCK and not the arena lock.  We never attempted to
>> fix that with #16159; our fix was limited to making malloc itself work
>> and we never got to any use cases beyond that.
> 
> Carlos had previously raised a question about user-supplied malloc
> implementations calling dlopen.  That might be the case that seemed most
> related to 16573.  But I don't recall if there was a resolution to that.

The bug is in mtrace, and is exactly as the submitter describes.
 
Foreign function hooks left in place when real malloc is called must be SR-safe [1].

The foreign function hooks implemented by malloc/mtrace.c are not SR-safe
and therefore are prone to deadlock on their own locks.

One way to fix this is to remove all foreign function hooks when calling the
real malloc (as submitter suggets, which is a good short-term solution).

The other way is to rewrite the foreign functions to be SR-safe and tolerate
being called again.

To answer your last question "Is it valid for a user-supplied malloc to call
dlopen?" The answer is "No." The community agreed that no blanket statement 
should be made about the SR-safety of any group of functions and that each
function should be evaluated and documented. As of today a user-supplied malloc
may only make AS-safe function calls to avoid potentially calling a library
routine reetrantly that cannot handle such an invocation. And none of this is
formally documented yet.

Cheers,
Carlos.

[1] https://sourceware.org/ml/libc-alpha/2014-12/msg00989.html
    https://sourceware.org/ml/libc-alpha/2014-12/msg00954.html