Re: glibc 2.3 fork() on i386-gnu crashes if malloc is overridden

[Wolfram Gloger <Wolfram dot Gloger at dent dot med dot uni-muenchen dot de>]

> bash uses its own malloc implementation.  Because of that, ptmalloc_init()
> is never called in bash, and the malloc internal variables like main_arena
> stay uninitialized.  In particular, main_arena.next is a null pointer.

Correct.

> Now, bash calls fork(), which runs the fork prepare handlers in the Hurd,
> among them ptmalloc_lock_all.  The implementation of that crashes if
> main_arena.next is a null pointer, because the for loop doesn't terminate.
...
> So, either ptmalloc_init must always be called for us, or the fork hook
> code needs to be made robust not to fail if malloc wasn't initialized,

I opted for this solution, see below.

> the hook functions shouldn't be registered if ptmalloc_init wasn't called,
> or another solution I can't think of.

It would be best if the atfork hooks wouldn't be registered at all if
a non-glibc malloc was in use, just like it is on anything but Hurd.
But I suspect this is impossible.

Regards,
Wolfram.

2002-11-18  Wolfram Gloger  <wg@malloc.de>

	* malloc/arena.c
	(ptmalloc_lock_all, ptmalloc_unlock_all, ptmalloc_unlock_all2): Do
	nothing if not initialized.  Bug report from Marcus Brinkmann
	<Marcus.Brinkmann@ruhr-uni-bochum.de>.

--- arena.c	2002/01/18 10:27:41	1.2
+++ arena.c	2002/11/18 12:32:06
@@ -73,6 +73,9 @@
 /* Mapped memory in non-main arenas (reliable only for NO_THREADS). */
 static unsigned long arena_mem;
 
+/* Already initialized? */
+int __malloc_initialized = -1;
+
 /**************************************************************************/
 
 #if USE_ARENAS
@@ -212,6 +215,8 @@
 {
   mstate ar_ptr;
 
+  if(__malloc_initialized < 1)
+    return;
   (void)mutex_lock(&list_lock);
   for(ar_ptr = &main_arena;;) {
     (void)mutex_lock(&ar_ptr->mutex);
@@ -232,6 +237,8 @@
 {
   mstate ar_ptr;
 
+  if(__malloc_initialized < 1)
+    return;
   tsd_setspecific(arena_key, save_arena);
   __malloc_hook = save_malloc_hook;
   __free_hook = save_free_hook;
@@ -255,6 +262,8 @@
 {
   mstate ar_ptr;
 
+  if(__malloc_initialized < 1)
+    return;
 #if defined _LIBC || defined MALLOC_HOOKS
   tsd_setspecific(arena_key, save_arena);
   __malloc_hook = save_malloc_hook;
@@ -275,9 +284,6 @@
 #endif
 
 #endif /* !defined NO_THREADS */
-
-/* Already initialized? */
-int __malloc_initialized = -1;
 
 /* Initialization routine. */
 #ifdef _LIBC