[PATCH] malloc: Do not corrupt top of threaded heap if top chunk is MINSIZE

[Mel Gorman <mgorman at suse dot de>]

mksquashfs was reported in openSUSE to be causing segmentation faults when
creating installation images. Testing showed that mksquashfs sometimes
failed and could be reproduced within 10 attempts. The core dump looked
like the heap top was corrupted and was pointing to an unmapped area. In
other cases, this has been due to an application corrupting glibc structures
but mksquashfs appears to be fine in this regard.

The problem is that heap_trim is "growing" the top into unmapped space.
If the top chunk == MINSIZE then top_area is -1 and this check does not
behave as expected due to a signed/unsigned comparison

  if (top_area <= pad)
    return 0;

The next calculation extra = ALIGN_DOWN(top_area - pad, pagesz) calculates
extra as a negative number which also is unnoticed due to a signed/unsigned
comparison. We then call shrink_heap(heap, negative_number) which crashes
later. This patch adds a simple check against MINSIZE to make sure extra
does not become negative. With it applied, mksquashfs ran successfully
1000 times without failure. There is no changelog update as it's a
follow-on from "malloc: Consistently apply trim_threshold to all heaps
[BZ #17195]".

---
 malloc/arena.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/malloc/arena.c b/malloc/arena.c
index 2466697d1aa7..b932574ccaf8 100644
--- a/malloc/arena.c
+++ b/malloc/arena.c
@@ -698,6 +698,9 @@ heap_trim (heap_info *heap, size_t pad)
   /* Uses similar logic for per-thread arenas as the main arena with systrim
      by preserving the top pad and at least a page.  */
   top_size = chunksize (top_chunk);
+  if (top_size == MINSIZE)
+    return 0;
+
   top_area = top_size - MINSIZE - 1;
   if (top_area <= pad)
     return 0;