[Bug libc/214] New: sbrk() doesn't detect brk() failures. Malloc doesn't handle sbrk() failures

["dlstevens at us dot ibm dot com" <sourceware-bugzilla at sources dot redhat dot com>]

On Linux systems, the "brk" system call returns 0 for success, but returns the
old end-of-data-segment when the RLIMIT_DATA soft limit is exceeded
(kernel 2.6.x).

"sbrk" in glibc has the following code:

       if (__brk (oldbrk + increment) < 0)
             return (void *) -1;
       return oldbrk;

On Linux systems, sbrk() returns the same value whether or not the allocation
exceeded the soft limit.

For Linux systems, this test should be:

        if (__brk (oldbrk + increment) != 0)
              return (void *) -1;
        return oldbrk;

I retested with a modified Linux kernel where brk() returns -1 for the failure
case and this led to another bug in malloc() [not yet filed or investigated].
Apparently, malloc() does not properly check for sbrk() failures in all cases,
so the retest resulted in a segmentation violation within the malloc code,
instead of malloc() returning NULL as expected.


Reproduce (sbrk issue) by:
     get current VmData size by reading /proc/self/status.
     use setrlimit() to set RLIMIT_DATA/rlim_cur to that value
     do an sbrk() of more than a page

Reproduce (malloc issue) by:
     same, but do enough malloc()'s to exhaust MMAPs; when it
     falls back to sbrk(), it gets a segmentation violation

-- 
           Summary: sbrk() doesn't detect brk() failures. Malloc doesn't
                    handle sbrk() failures
           Product: glibc
           Version: 2.3.2
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: gotom at debian dot or dot jp
        ReportedBy: dlstevens at us dot ibm dot com
                CC: glibc-bugs at sources dot redhat dot com


http://sources.redhat.com/bugzilla/show_bug.cgi?id=214

------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.