[Bug malloc/20646] New: sysmalloc incorrectly fails with custom morecore function

["guillaume at morinfr dot org" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=20646

            Bug ID: 20646
           Summary: sysmalloc incorrectly fails with custom morecore
                    function
           Product: glibc
           Version: 2.19
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: malloc
          Assignee: unassigned at sourceware dot org
          Reporter: guillaume at morinfr dot org
  Target Milestone: ---

Created attachment 9535
  --> https://sourceware.org/bugzilla/attachment.cgi?id=9535&action=edit
small program to reproduce the bug

I am reporting this bug against 2.19 because I am testing on Debian jessie but
I am pretty sure the bug still exists in the git master.

This is a bit of an obscure bug.  If you switch main heaps and end up with some
non-continuous addresses, sysmalloc will try to free what's left of the old
heap:

line 2656:
                     /* If possible, release the rest. */
                      if (old_size >= MINSIZE)
                        {
                          _int_free (av, old_top, 1, 1);
                        }

However if you have switched to a different morecore() (say from brk() to your
own custom morecore()), the current implementation has no idea you're trying to
shrink a different heap and calls morecore(-value) which shrinks the *new* heap
containing the freshly allocated memory.

At that point, the if statement checking is the allocation succeeded will fail
because the wrong heap was shrunk despite the fact that morecore() worked just
fine earlier.

In general, you cannot observe it because if _int_malloc() fails, the default
malloc() tries to use an arena and this generally succeeds.  To reliably
reproduce it though, you can use MALLOC_CHECK_ on the command line.  The code
only tries to call _int_malloc(), so the bug can be reproduced every single
time.

I am attaching a program which reproduces the problem.  You'll have to mkae
sure you run it with MALLOC_CHECK_ set, e.g
$ MALLOC_CHECK_=2 ./malloc_shrink_morecore
Failed!
$

This might seem esoteric but this happens when using libhugetlbfs to use huge
pages for the heap.  If you enable shrinking (disabled by default in
libhugetlbfs), suddenly allocations start to fail (esp if you run with
MALLOC_CHECK_)

One fix is to disable calling systrim() in the _int_free call mentioned above. 
I am also attaching a patch doing just this but I am not familiar with the
malloc internals so you might prefer a different fix.  It applies on glibc 2.19
if you need something against the master, let me know.

Thanks in advance,

Guillaume.

-- 
You are receiving this mail because:
You are on the CC list for the bug.