Re: An approach to segregated linking using dlmopen

[Vivek Das Mohapatra <vivek at collabora dot com>]

As I've mentioned here before I've been working on a segregated
library helper based on dlmopen: The goal is to allow applications
(games are the current focus) to use the system's libGL and friends
without being exposed to any extra dependencies they would bring in
while still having them run from a restricted runtime (with, for
example, a set-in-stone list of libraries at specific versions).

I have it largely working -

The code (and a quick set of instructions to try it) is here:

  https://gitlab.collabora.com/vivek/libcapsule

And I gave a talk about it (if such things interest you) recently here:

  https://wiki.debian.org/DebianEvents/gb/2017/MiniDebConfCambridge/Dasmohapatra

Almost all of it works using just public glibc interfaces (and
/proc/self/maps if we bump into RELRO linking), except for one thing:

Since the dlmopen namespace gets its own instance of libc, it has its
own *alloc/free cluster of functions and its own arena(s) from which
to allocate memory: Memory allocated from inside the namespace must be
freed by the matching free function.

It turns out that this is not usually a problem (for me) - allocation
and freeing does not straddle the namespace barrier for most things
I've tried so far, but in the interests of making the library more robust
I'd like to solve this problem properly (and cleanly).

My first attempt to solve the problem was to [re]use the symbol hijacking
on which libcapsule is based to make the new namespace use the same
malloc/calloc/realloc/posix_memalign/free cluster as the default namespace.

This mostly works, but it seems that a few pointers are either allocated
too early for me to intercept, or they are allocated by some other mechanism
or entry point that I've missed.

The next step was to provide wrappers for free and realloc, installed
in the new namespace (as above) which try to detect which implementation
the pointer came from, and pass it there for freeing/reallocing.

This does seem to work, but is at best a temporary hack as it requires
some knowledge about the magic memchunkptr size bits and a bunch of
private glibc macros (it lets the rest of the project advance and stops
me being a blocker so it'll do for now but I don't want to release with
these hacks in place).

It's also not clear to me if there's any way of determining _which_
malloc/free cluster was responsible for a mmap()ed allocation.

I've been thinking about this for a while, and it seems to me there are
a couple of reasonable approaches to fixing this:

 - Implement a new flag, RTLD_UNIQUE or similar, which tells dlmopen
   to use the existing link-map entry for the target DSO in the new
   private link map.

   Since I manage each dlmopen() individually in libcapsule, I could
   open the core libc DSOs with this flag, and guarantee I had exactly
   one malloc/free cluster

 - Expose the free/realloc pointer sanity checks via public API:
   I wouldn't need the details: A level of "yes, this pointer is
   mine" vs "nope, this wasn't from me" would do the job.

 - Something else I haven't thought of?

I think the first approach is the cleanest and almost certainly the
most efficient - If the idea of such a flag or the behaviour it implies
is unacceptable, the second would also be Ok (it's basically what I do
now except I have to do it by stealing a bunch of private glibc macros).

Thoughts?