Re: [PATCH] [RFC] malloc: Reduce worst-case behaviour with madvise and refault overhead

[Mel Gorman <mgorman at suse dot de>]

On Thu, Feb 12, 2015 at 06:58:14PM +0100, Julian Taylor wrote:
> > On Mon, Feb 09, 2015 at 03:52:22PM -0500, Carlos O'Donell wrote:
> >> On 02/09/2015 09:06 AM, Mel Gorman wrote:
> >> > while (data_to_process) {
> >> > 	buf = malloc(large_size);
> >> > 	do_stuff();
> >> > 	free(buf);
> >> > }
> >> 
> >> Why isn't the fix to change the application to hoist the
> >> malloc out of the loop?
> > 
> > I understand this is impossible for some language idioms (typically
> > OOP, and despite my personal belief that this indicates they're bad
> > language idioms, I don't want to descend into that type of argument),
> > but to me the big question is:
> > 
> > Why, when you have a large buffer -- so large that it can effect
> > MADV_DONTNEED or munmap when freed -- are you doing so little with it
> > in do_stuff() that the work performed on the buffer doesn't dominate
> > the time spent?
> > 
> > This indicates to me that the problem might actually be significant
> > over-allocation beyond the size that's actually going to be used. Do
> > we have some real-world specific examples of where this is happening?
> > If it's poor design in application code and the applications could be
> > corrected, I think we should consider whether the right fix is on the
> > application side.
> > 
> 
> 
> I also ran into this issue numerous times, also filed a bug:
> https://sourceware.org/bugzilla/show_bug.cgi?id=17195
> 

Thanks for pointing that out. I read the report and the original report
and do not understand why it was considered a duplicate. They are
completely different issues.

> As a real world example I have higher level numerical software.
> E.g. in python numpy you write code like this:
> a = b+ c +d
> where these are large arrays due to limitations of the library and
> python this involves allocating multiple large arrays while the
> operations on the memory itself is very small.

Is there any chance you could supply a simple test case in python for
this? Your description is straight-forward and I suspect the resulting
script will be just a few lines long but I want to be sure I see the
same problem.

Alternatively, would you be in the position to test v2 of this patch and
see if the performance of your application can be adressed by tuning trim
threshold to a high value?

Thanks.

-- 
Mel Gorman
SUSE Labs