Re: RFC: api to walk memory allocations

[Rich Felker <dalias at aerifal dot cx>]

On Thu, Oct 31, 2013 at 02:49:54PM -0600, David Ahern wrote:
> 
> On 10/30/13, 9:31 PM, Rich Felker wrote:
> >On Wed, Oct 30, 2013 at 09:09:02PM -0600, David Ahern wrote:
> >>One scenario here is a customer suspects a memory leak in a process
> >>and runs a CLI command that essentially walks the memory data
> >>structures. With the callback we can run a sanity checker or what
> >>have you. There's more to it from an infra perspective, but that
> >>gets into ap design. glibc wise it is a matter of walking glibc data
> >>structures and allowing aps to have a means of sanity checking.
> >
> >I'm not clear how much that really buys you. To debug the leak you
> >need a lot more information about the owners of the allocations, not
> >just the map of all existing allocations.
> 
> Yes, there is more to the design, but it is irrelevant to the glibc
> request so I did not include it. We have infrastructure to track
> each malloc/free. It is fairly trivial to hook malloc/free and add a
> footer/header to each allocation. The meta-data here contains
> whatever we feel is sufficient to track the source of the
> allocations. The part up for debate is efficiently walking the
> allocations without maintaining yet another data structure to do it.

If you really want to do this, "yet another data structure" is the
right way to do it. Locking in a dependency on current implementation
details is definitely the wrong way to do it.

> >The problem with "leveraging existing data structures" is that it
> >locks you into the current existing data structures. Building as a
> >separate layer, or separate malloc implementation you could link in,
> >may cost more for users of this feature, but avoids the perpetual cost
> >for everyone else who doesn't need it.
> 
> The point of building something into glibc is that our code does not
> have to have knowledge of malloc internals. I can be rather dense at
> times, so can you elaborate on how a memwalk api restricts the
> malloc implementation? The whole point here is that nothing about
> the internal implementation is exposed. The memwalk API is simply
> pointer, length and a callback into the app. None of that involves
> the app knowing anything about how malloc works.

The problem is that it requires glibc to commit permanently to keeping
the data structures needed to walk the internals in the the memwalk
API allows, even if keeping those structures is otherwise unnecessary,
and even if more efficient (in storage or performance) implementations
can be had by omitting this information.

Rich