This is the mail archive of the libc-help@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

RE: Possible malloc/mmap bug?


> Your application is a pathological case of VM fragmentation.


Oh, I know. :)

But in a 64 bit operating system w/ the amount of virtual address space in the Exabyte range, and the fact that while we are processing large amounts of data (pre-reduction) and us preparing batches to feed to the program that won't exceed (pre-reduction) the system RAM (48GB), it's not an issue here.


> Well before you reach M_MMAP_MAX on any reasonable system you'll get
> ENOMEM from mmap.


Not on the reasonable box I described above. :)  Or my dev box (64 bit, 24GB) or the 2 VMs I created (64 bit system but only 2GB of RAM).  The test case runs until the default M_MMAP_MAX on all of them, where it then fails.


> On a local small 32-bit system I see the failure at ~22,000 iterations:
> ~~~
> 1321628743.652568 mremap(0xbff7e000, 532480, 8192, MREMAP_MAYMOVE) =
> 0xbff7e000
> 1321628743.652650 mmap2(NULL, 532480, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> 1321628743.653300 brk(0)                = 0x96f4000
> 1321628743.653369 brk(0x9796000)        = 0x96f4000
> 1321628743.653442 mmap2(NULL, 1048576, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
> ~~~
> I have a glibc built with debugging and the failure mode is as follows:
> * The mremap is the realloc happening.
> * The mmap2 fails with ENOMEM since the kernel has run out of space
> for any mappings.
> * Glibc then tries brk which *also* fails. Notice the returned address
> is unchanged indicating no more space is left.
> 
> At this point the VM is entirely fragmented and the current glibc
> malloc implementation can't recover.


I'm pretty sure that this is a completely different failure than what I am seeing, as I have plenty of RAM and Virtual Address space left at the time of failure...


> You *could* argue that this is a bug in the glibc malloc
> implementation, but the solution (compacting the allocations during
> realloc) would slow down other well behaved applications.


Naw, the failure you see and describe makes sense to me, I wouldn't consider that a malloc bug.


> You need to rewrite your application to:
> * Allocate and maintain a single working buffer (or one per thread).
> * When you complete the work on the working buffer copy the results to
> a newly allocated small buffer.
> * Reuse the single working buffer.
> 
> This will prevent fragmentation and also have the effect of making
> your application faster.


Actually, the reason my coworker wrote the code the way he did was in the interest of several aspects of performance.  We are processing around 10TB of data through this system (in carefully predetermined batches to help optimize the results) and it's going to take 30-60 days at least...any increase in the processing time can result in adding days or weeks to the overall processing.

That having been said, I am contemplating writing a new version of it that would do some of the processing in parallel and might possibly take the same amount of time or maybe a little less, and be able to free up RAM as it goes along.

But that is neither here nor there...I still think that the failure I'm seeing is different than what you described above, and possibly a legit bug.  And an especially interesting one given the fact that static linking seemed to eliminate the problem.

I wish I could have successfully created the shared libc.so from the stock glibc source I was hand building so I could verify that it's truly an issue there too (and have an environment where I could try and debug it myself and perhaps submit a patch). :)

You don't by any chance have a 64 bit glibc development environment you could try my test case on, do you?

Thanks,
Chuck




Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]