Bug 1128

This bug can be addressed with a trivial one liner patch, which is not even in 
C code proper but a preprocessing directive!

ROFL!!!

I verified this to work on a MIPS system based on kernel 2.6.17.7 and glibc 
2.5.

The system boots just fine, everything seems to run, and my malloc test case 
shows that   1)  memory is being returned to the OS by free even after sbrk 
fails, and 2) sbrk is not repeatedly called after it fails. (The test case 
simply allocates about 500 megs worth of memory in 500 byte pieces). Malloc 
nicely switches to an alternate arena, whereby it uses mmap to grab heaps. 
These heaps are nicely freed when they become deallocated, and the memory 
originally obtained from sbrk is nicely returned with a negative sbrk. I ran 
it under gdb, placing breakpoints on mmap, munap, and sbrk, and also watching 
the virtual memory size of the process using ps aux.

--- glibc-2.5.orig/malloc/malloc.c      2006-09-07 09:06:02.000000000 -0700
+++ glibc-2.5/malloc/malloc.c   2008-04-04 01:01:32.862714704 -0800
@@ -3056,15 +3056,24 @@
       (*__after_morecore_hook) ();
   } else {
   /*
-    If have mmap, try using it as a backup when MORECORE fails or
-    cannot be used. This is worth doing on systems that have "holes" in
+    If we have arenas, then we are basically done with this
+    main_arena. We can return null, because the higher level malloc
+    routine like public_mALLOc will try fetching from another
+    arena, creating one if necessary. This way we won't
+    cause the main_arena to become non-contiguous.
+    A non-contiguous main_arena has the bug that it won't return
+    memory to the OS when it is freed, and that it keeps
+    hammering away on a nonworking sbrk for each allocation request
+
+    If have mmap, but not arenas, try using it as a backup when MORECORE fails
+    or cannot be used. This is worth doing on systems that have "holes" in
     address space, so sbrk cannot extend to give contiguous space, but
     space is available elsewhere.  Note that we ignore mmap max count
     and threshold limits, since the space will not be used as a
     segregated mmap region.
   */

-#if HAVE_MMAP
+#if HAVE_MMAP && !USE_ARENAS
     /* Cannot merge with old top, so add its size back in */
     if (contiguous(av))
       size = (size + old_size + pagemask) & ~pagemask;

That would fix the test case in question, but only because it prohibits
allocation from the main arena altogether in this specific case.

I think it would not fix the following (IMHO in practice more likely) situation:

1. lot's of malloc/free from the main arena
2. a single sbrk(+something) call from the app, making the main arena
discontiguous after just one additional malloc()
3. lot's of free() from the main arena -- now every free() can still incur an
sbrk(0) call because the end of the main arena never can pass the user-allocated
space

I think the proper fix would be to prevent or alleviate the sbrk(0) in every
free(), given that sbrk(0) is very expensive (but see below).
Maybe we should give up expecting a discontiguous main_arena to magically become
contiguous again, i.e. not call trim at all when the arena is marked noncontiguous.

BUT, taking a step back, is this really a serious problem?  I consider the
statement that "each call to free() incurs a system call" _false_, because libc
caches the last brk value, so sbrk(0) does _not_ perform a system call AFAICS.
Or are we talking about specific systems where sbrk(0) is just like sbrk(+x)?

(In reply to comment #2)
> I think it would not fix the following (IMHO in practice more likely) 
situation:
> 1. lot's of malloc/free from the main arena
> 2. a single sbrk(+something) call from the app

IMHO, apps that call sbrk deserve less than optimal behavior from malloc, such 
as memory not being returned when it is freed.

> 3. lot's of free() from the main arena
> -- now every free() can still incur an
> sbrk(0) call because the end of the main arena never can pass the user-
allocated
> space
> I think the proper fix would be to prevent or alleviate the sbrk(0) in every
> free(), given that sbrk(0) is very expensive (but see below).

But, seeing below, it is successfully argued that sbrk(0) is actually not 
expensive at all thanks to cashing. The remaining problem is that the freed 
memory won't be returned to the operating system.  But the application 
arguably deserves that, because it interfered with malloc by calling sbrk
(+delta).

I reported this bug about three years ago now, so my memory is a little bit hazy. However, I do remember 
that calls to free() were definitely causing expensive syscalls. I note the recent comments about sbrk(0) 
not being expensive due to caching and, reviewing the glibc sources, I have to agree with this. So I suspect  
there may be some detail missing from my original report that explains more what was going on.

Sorry!

Created attachment 4242 [details]
test case

Is there any real solution for this problem? The "mallopt(M_TRIM_THRESHOLD,
(unsigned long) -1)" workaround does not work for me. It just causes every
malloc/free to result in a mmap/munmap syscall, which is even slower.

I have attached a small test program which can (kind of) work around the
problem by placing an explicit 'block buffer' behind the work buffers (set
'avoid_intelligence = true'). This would be hard to achieve in a real-world
program, though.

system info:
glibc 2.10.1-4
$ uname -a
Linux myhost 2.6.31-bfs #1 SMP PREEMPT Mon Sep 28 14:26:47 UTC 2009 x86_64
Intel(R) Core(TM)2 Duo CPU P9400 @ 2.40GHz GenuineIntel GNU/Linux

The strace output on the test program is:
brk(0xcab000)				= 0xcab000
brk(0xc2b000)				= 0xc2b000
brk(0xc6b000)				= 0xc6b000
brk(0xcab000)				= 0xcab000
(... continued ...)

oops, misinterpreted the topic of this bug:
my previous test case is for the Problem
'Malloc trimming filled-up sbrk space too aggressively causes unnecessary system
call overhead'
It is still a serious problem (caused about 50% slow down in a real world
program. The only possible work around was to introduce much more complicated
memory management in my application)

In reply to comment #5:

The test program you've posted does potentially cause lots of system calls, but
for a different reason than the original report:
You're allocating 256k blocks which exceed the default mmap threshold.
So, to avoid any mmapping in that kind of app you have to specify _both_

mallopt(M_TRIM_THRESHOLD, (unsigned long) -1);
mallopt(M_MMAP_THRESHOLD, (unsigned long) -1);

Does following patch fixes performance for you? I want to be sure that problem was in sbrk but not in something else.

diff --git a/malloc/malloc.c b/malloc/malloc.c
index 2938234..ab6d46f 100644
--- a/malloc/malloc.c
+++ b/malloc/malloc.c
@@ -2716,6 +2716,10 @@ static int systrim(size_t pad, mstate av)
   char* current_brk;     /* address returned by pre-check sbrk call */
   char* new_brk;         /* address returned by post-check sbrk call */
   size_t pagesz;
+  static int give_up;
+
+  if (give_up)
+    return 0;
 
   pagesz = GLRO(dl_pagesize);
   top_size = chunksize(av->top);
@@ -2762,7 +2766,8 @@ static int systrim(size_t pad, mstate av)
          return 1;
        }
       }
-    }
+    } else
+      give_up = 1;
   }
   return 0;
 }

*** Bug 17195 has been marked as a duplicate of this bug. ***