This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [patch] Make the mmap/brk threshold in malloc dynamic to improve performance


Wolfram Gloger wrote:
Hello,

Mmaping to allocate memory is a loss if we are going to free it and
reuse it, because we have to do this clearing unnecessarily.

This can be true or false -- if you are going to free a large chunk and then re-allocate it hours later the clearing will not matter much compared to the amount of memory saved (to be multiplied by the amount of time during which the chunk was available to other threads or applications).

well the VM subsystem is also not stupid and will reclaim over time as well


The real question Arjan and I have is what are the conditions under
which allocating using mmap is a win?  Here's what I can think of:

* When we're running out of brk() space - when does this happen?

It will happen some time, and then it was "nice" to be able to fall back to mmap().

we're not arguing to remove mmap at all; just to shift the usage threshold of it ;)


* When it will reduce fragmentation in the brk()'d memory

This one was and is the real issue, and was the original reason for introducing mmap(). Note that fragmentation is always leading to at least linear effects: the larger the amount of memory allocated, the more memory is wasted by fragmentation. So, the fact that we are now having much more RAM and driving much larger working sets than before does _not_ invalidate striving for less fragmentation. In fact, the opposite should be the case.

but the points where fragmentation starts is different; the size of long lived allocations (because size is only used as heuristic to separate long lived and short lived) is bigger now.


And don't assume mmap doesn't fragment by nature. It fragments just as much, if not more....


It seems like having larger address spaces is obviating most reasons
for using mmap.

I disagree, obviating address space limits affecting brk() was just a very minor reason for using mmap(). The main reason was conserving memory -- not a fixed amount but a proportion of the working set.

I think this is false economies. The kernel VM is a reclaiming VM, and to be honest, a bit of hysteresis is ok. glibc could even decide to shrink the hysteresis window over time when there is no activity. But that I think is just too fancy ;)




Making the mmap threshold dynamic sure seems like an interesting idea
to me.  However, the original patch posted by Arjan would IMHO only
quickly raise the threshold up to the maximum, and never decrease it
again.  Since mmap() only comes into play for large allocations, it
could well be worthwhile to track allocation times for these chunks
and adjust the threshold based on the tendency to _quickly_ free() and
re-allocate them.

I think the better quick fix would just be to raise the default mmap
threshold like it was done for the trim threshold some years ago.  The
default mmap and trim thresholds could also be set as a fraction of
available RAM (between sane limits of course).  Do we have
sysconf(_SC_PHYS_PAGES) available already at time of the initial
malloc() call?


that may not be useful, may I suggest a percentage of virtual address space?



Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]