Re: Persistent malloc

[Carlos O'Donell <carlos at redhat dot com>]

On 11/08/2017 06:26 AM, Adrian May wrote:
>> We are going to remove the MORECORE hook soon.
> 
> If that's because it seems odd for the main heap to work differently from
> the others, then I see the point. But before the excellent multiple-heaps
> thing came along, MORECORE was a pretty good idea. Is there a plan to
> deploy something with the same virtues?

No. You will have one of two choices:

* Use the system allocator as-is with the tunables that are present
  (which allow tweaking the behaviour in significant ways).

* Provide your own malloc API implementation, and all the interface
  that this needs, via ELF symbol interposition e.g. LD_PRELOAD.

We will not be providing a way to interchange just the low-level memory
sources e.g. sbrk, mmap.

> It was clever of them at the time to see that malloc comprises a whole lot
> of tricky free-chunk juggling that was likely to develop into a very
> detailed science, but that such an algorithm applies to ANY linear resource
> that gets extended from one end. So it would be a shame to take all that
> science and strongly couple it to just one kind of linear resource. The
> fact that I could make the heap persistent with so little work is testimony
> to that foresight, and if the plan for the future is that people have to
> more-or-less copy-paste malloc.c, well that would be a step backwards in my
> mind. Nevertheless, polishing away the difference between the main and
> subsequent arenas does make sense.

It is *absolutely* a step backwards for *your* use case. However, you have to
consider the global set of use cases, of which yours is just one, and the cost
of that support applies globally.

> I guess what you have planned for the future is a pure mmap-based solution,
> and in that case it would seem even easier to do what I did cos I'm using
> mmap as well. Now the only difference is that I have a file under the mmap
> and would like to flush it from time to time. As long as I get told which
> heap it is, I'd be happy to have multiple files for multiple heaps.

I would like to see the main arena distinction go away, but I expect developer
tooling expects the kernel's heap to be used via sbrk, so removing it might
cause some compatibility problems. So while the internal data structures will
become more similar for the main and non-main arenas, the main arena may
continue to use sbrk to get memory (until it fails and switches to mmap).
 
> Is that what you all had in mind, or are you just dropping the reasons why
> MORECORE was invented?
MORECORE is an unencrypted function pointer which is an attack vector to
hijack program allocations. It is a security problem.

On top of this it is a hook into malloc which impedes the functional changes
we want to make to the internals.

All of these things, coupled with the almost zero real-world use cases means
this has high cost with little reward.

We are removing it so we can simplify and improve glibc's malloc.

-- 
Cheers,
Carlos.