Re: TLS redux

[Rich Felker <dalias at aerifal dot cx>]

On Tue, Jan 14, 2014 at 06:23:35PM -0800, Roland McGrath wrote:
> I've finally caught up on the long threads about TLS issues.
> (The good news is that this was a sizable fraction of all of my
> libc-related backlog, so I'm much less behind than I was before!)
> 
> Other people have discussed many of the issues that I would have
> raised if I'd participated all along, but not all of them.  I won't
> summarize the whole discussion, but just mention the things I think
> it's important not to overlook.  I don't really have anything to say
> about most of the implementation details.  Only the last point or two
> are issues about the changes being considered for 2.19.
> 
> * Lazy allocation is an explicit feature of the TLS ABI, not an
>   incidental detail.  The wisdom of the feature can be debated, but
>   the compatibility requirements are clear.

Yes, yes, and yes. :-)

>   It's a regression if this scenario stops working:
>   1. Start a thousand threads
>   2. dlopen a module containing __thread char buf[100 << 20];
>   3. Start another thousand threads
>   4. Call into the module on one thread so it uses its buf.
>   5. Start a third thousand threads
>   Now you should have 3000 threads but not 3000*100M memory use.
>   (Here I mean address space reservation, regardless of consumption
>   of machine resources, VM overcommit, etc.)
> 
>   At least in the case of an existing binary dlopen caller (which
>   could actually be either in an executable or in a DSO) and an
>   existing binary module loaded by that dlopen, such a regression is
>   an ABI break and cannot be tolerated.

I don't see how dlopen failing, and reporting the failure, could be an
ABI break. It may be a "feature" regression, but the ABI contract is
not broken.

> * It's inherently impossible to both allocate lazily and have dynamic
>   TLS access that cannot fail.

I think that's been obvious all along.

>  Either you preallocate the memory
>   (eager use of address space, if not necessarily actual storage) or
>   attempting to allocate it later might fail.  Hence it must be an
>   explicit choice between the two.  That choice might be at the
>   granularity of the whole implementation, as in musl, or all the way
>   down to the granularity of an individual TLS-containing module or
>   individual module-loading call.  Since glibc has a compatibility
>   requirement to support lazy allocation, the only possibilities for
>   the contrary choice are at smaller granularities.

My preference would be for the granularity to be the symbol version
level, i.e. deprecate lazy allocation for newly-build applications.
For 64-bit targets, I hardly even consider this a regression; if you
have overcommit enabled in the kernel (which is still the default),
allocations are not going to fail for lack of physical storage, and
exhausting virtual address space before you exhaust thread id space
(kernel tids are either 29- or 30-bit; I forget which) is virtually
impossible.

However I understand that others may want a finer granularity.

> * Eager allocation could be a new option, and could even be a new
>   default.  (What the default should be is a separate debate that does
>   not need to begin now.)

If it's an option, I would prefer it to be default. The main
motivation of this preference is to discourage developers from
unintentionally making DSOs that fail without lazy allocation.

> ** e.g. A new DF_1_* flag and -z option for a DSO to request it.
> *** Could be made default for newly-built DSOs.
> ** New dlopen flag bits to request it.
> *** Could be made default for newly-built dlopen callers (i.e. new
>     symbol version of dlopen).

These sound like good ideas to me. One question is how it's applied to
libraries loaded recursively via DT_NEEDED, and if so, whether the
pre-allocation status of a library already loaded can be "upgraded"
when a newly-loaded library with an explicit dlopen flag depends on it
(and further, which direction of change would be considered an
"upgrade").

> * In implementing eager allocation when multiple threads already
>   exist, it is theoretically possible to do all or almost all of it
>   asynchronously (i.e. all work done inside the dlopen call on the
>   thread that called it).  It's trickiest, or perhaps impossible, to
>   do the final step if the DTV needs to be expanded, from another
>   thread.  But there is not really any good reason to do a lot from
>   other threads.  Rich Felker described the most sensible
>   implementation strategy: do all the allocation in dlopen, but only
>   actually install those new pointers synchronously in each thread,
>   inside __tls_get_addr.

One issue that might need some more consideration is freeing of the
memory. My implementation in musl doesn't have to worry about that
(although it does "leak" some memory if lots of threads were running
at dlopen-time and then exit) because dlopen is permanent (dlclose is
a no-op). But on glibc you want to ensure that both assigned and
unassigned eagar allocations get freed at some point if the DSO is
closed or the thread exits; otherwise there are long-term
dlopen/dlclose scenarios that will leak memory over time.

I don't remember right off if this was discussed at all so far.

> * The main request for async-signal-safe TLS use is satisfied by "fail
>   safe" semantics that preserve lazy allocation semantics: if the
>   memory is really not available, then you crash gracefully in
>   __tls_get_addr.  (That is, as much grace as abort, as opposed to the
>   full range of "undefined behavior" or anything like deadlock.)

I don't see how this is affected. "Graceful" crash is trivial to
provide.

> * How to find all memory containing direct application data is a de
>   facto part of our ABI.  By "direct" I mean objects that the
>   application touches itself.  That includes __thread variables just
>   as it includes global, static, and auto variables.  It excludes
>   library-maintained caches and the like, but includes any user data
>   that the public API implies the library holds onto, such as pointers
>   stored by <search.h> functions.
> 
>   This is a distinct issue from the general subject of "using an
>   alternate allocation mechanism for memory" that Carlos mentioned.
>   If libc changes how and where it stores its own internal data, that
>   does not impinge on anything that is a de facto part of the ABI.  If
>   libc changes how and where it stores application TLS data or other
>   things in the aforementioned category, that is another thing entirely.
> 
>   I mentioned ASan as just one example of the kinds of things that
>   might care about these aspects of the de facto ABI.  Things like
>   ASan and conservative GC implementations are the obvious examples.
>   But the fundamentals of conservatism dictate that we not make a
>   priori assumptions about what our users are doing and what matters
>   to them.  As with all somewhat fuzzy aspects of the ABI, there will
>   be a pragmatic balancing test between "I was using that, you can't
>   break it!" and, "You were broken to have been relying on that."  But
>   we must consider it explicitly, discuss it pragmatically, and be
>   circumspect about changes, especially the subtle ones.  The change
>   at issue here is especially subtle in that it could be a silent time
>   bomb that does not affect anybody in practice (or that nobody
>   realizes explains strange new flakiness they experience) for
>   multiple release cycles.  For example, if before the change a
>   __thread variable (in a dynamic TLS module) sometimes was the only
>   root holding a GC'able pointer and the GC noticed it there, but
>   after the change the GC doesn't see that root.  If this bug is
>   introduced tomorrow, it could be a long time before the confluence
>   of when collections happen, whether other objects hold (or appear to
>   hold) the same pointer, and the effects of reclamation, add up to
>   make someone experience a failure they notice.
> 
>   How to find threads' stacks and static TLS areas is already
>   underspecified (improving that situation is a subject for another
>   discussion).  But even for that, we would be quite circumspect about
>   making a change that could break methods existing programs are using
>   to acquire that information.

Could you elaborate on what methods existing programs are using?

>   Today, dynamic TLS areas are allocated using the public malloc
>   interface.  Programs or GC libraries or whatnot can today supply a
>   malloc replacement, scan the static data+bss area, scan each
>   thread's stack and static TLS area, and reliably discover every byte
>   of user data in any TLS area for any thread.  We never documented an
>   explicit guarantee that this works, but it does and to the extent
>   that anything extant relies on it (whether or not its maintainers
>   realize they do!), it is part of the de facto ABI.
> 
>   I don't have a firm conclusion about what we guarantees of this sort
>   we should or should not be offering or preserving.  But this change
>   affects that part of the de facto ABI and as far as I noticed nobody
>   has discussed it at all.  That fails the conservatism test instantly.

My perception (or at least my hope) is that glibc is adopting a
somewhat saner view of what being conservative means: moving away from
what I'd call the DOS/Windows approach, where any undocumented
internal interface any application developer might ever have
discovered and (ab)used is considered part of the ABI, and towards an
approach where there are well-defined public APIs and ABIs. (As an
example, deprecating the public __secure_getenv symbol comes to mind.)
Clearly it's hard for a project like glibc that has so many poorly
written legacy apps (at both the source and binary level) depending on
it to make this transition fully, but my view is that, insomuch as
it's practical, this is a very positive transition to be making.

Of course this will mean some bad applications break. I think it's a
very reasonable goal to aim for having as much breakage as possible
being detectable, and avoiding situations where widely used bad
practices that "used to work" turn into runaway UB. In particular,
using your example, efforts should be made to ensure that bad GC
doesn't miss TLS roots; however, if the only way to ensure this is by
keeping the implementation exactly how it is/was in the past, a better
alternative might be simply detecting the broken code and aborting.

> I have no great quarrel with the thoroughness or conservatism of the
> vetting of the implementation details or first-order ABI issues of
> what's gone in.  (I am not entirely sanguine about all that, but close
> enough that I've decided not to participate in the detailed review.)
> But the mere fact that in a few months a >100 messages of discussion,
> I'm the first to raise these subtleties (that I really thought would
> have been fairly obvious to people here) gives me great pause about
> the whole endeavor.

I don't think you're the first; most of the issues you raised are
things I considered obvious, and I thought they were at least
mentioned. The matter of GC roots and ASan might not have been
covered, but in fairness, it's not reasonable to expect everyone to be
an expert on every way some third-party software is abusing glibc
internals. One thing that would be nice to come out of this would be
if we could arrive at some sort of friendly procedure for third-party
projects wanting/needing this kind of poking-at-internals access to
contact the glibc team, explain the need, and work out whether there's
any way a reasonable public interface can be provided. Even if they
couldn't use the public interface immediately (e.g. need to support
old versions in the wild), they could at least have their software
ready to use public interfaces for the future, so that their "poking
at internals" would only need to be compatible with a finite set of
past releases rather than an "infinite set" of future releases.

> Similarly, Carlos expressed an attitude that I'll summarize as, "So we
> break ASan for a release or three and fix it later, no big deal."
> That is fundamentally anti-conservative IMHO.  Indeed, ASan is not
> part of glibc.  If it were, we'd be able to achieve complete
> confidence about all its issues very quickly.  ASan is an example of
> the wide variety of things users are doing with glibc, that we have an
> obligation never to break silently or inadvertently.

I'm not an expert on ASan and even what its level of maturity is right
now, so I'm going to hold off on going into the topic yet.

> Dynamic TLS access not being async-signal-safe has been the status quo
> since the inception of the TLS features.  Leaving that as it is for
> another release is just obviously acceptably conservative.
> Contrarily, breaking other kinds of subtle interaction with TLS
> features that have worked in practice heretofore is not conservative
> at all.

If TLS access from signal handlers always failed in the past, I would
agree with your assessment that this was the "status quo". But the
reality is that failure of something to be AS-safe is not observable
except as race conditions which, depending on usage patterns, may be
extremely rare to hit. So in my opinion, failure of TLS access to be
AS-safe, when it was never documented as non-AS-safe, is not a known
"status quo" but a bug, and a possibly one that's very difficult to
track down and work around.

One change in glibc policy I'm very happy about, and don't want to see
a regression on, is abandoning the attitude that bugs can't be fixed
because the fix might break existing applications that depend on those
bugs.

> As I said, I'm not specifying any conclusions.  I'm fairly confident
> we can find a middle road that is appropriately conservative while
> offering improvement for the pain point.  But we have yet to even
> begin discussing what IMHO should be considered a major obstacle to
> making this change while keeping with our conservative principles.

I agree too. I'd like to hear more on whether there are aspects of the
current changes so far that you think are actually going to break
third-party software (as opposed to just having the as-yet-unstudied
theoretical potential to do so), and any further ideas (beyond the
flags/versioning/etc. for dlopen and DSO ELF headers) you might have
for improving the situation.

Rich