Re: proposal: Thread Properties API

[Konstantin Serebryany <konstantin dot s dot serebryany at gmail dot com>]

Let me try (mostly reformatted the wiki page)...

============== Requirements ===================
Various dynamic program analysis tools may need to query glibc about
various properties and get notifications about events of the current
thread:
Queries:
   - stack boundaries (for non-main threads)
   - static TLS boundaries
Notifications:
   - dynamic TLS creation/destruction
   - thread death (if possible)

<details>
AddressSanitizer (aka ASAN) is a tool that detects use-after-free,
heap- stack- and global- overflows, and other bugs. Available in Clang
(starting from 3.2) and GCC (starting from 4.8).
ASAN needs to know the stack and tls boundaries to properly report
error messages.

LeakSanitizer (aka LSAN) is a memory leak detector that can work as a
separate library or bundled with AddressSanitizer. Available in Clang
(starting from 3.4) and GCC (starting from 4.9).
LSAN needs to know all stack and tls boundaries for all living threads
in order to avoid false positive leaks reports. At the beginning of
the leak detection phase (typically, at the program shutdown), LSAN
creates a separate process that shares the address space with the main
process, then attaches to all threads with ptrace. At this point LSAN
should be able to extract properties of all threads (such as stack and
tls bounds) w/o relying on libc functions, which means that all thread
properties should already be stored somewhere in LSAN's own data
structures.
Example: test case where not knowing about dynamic TLS leads to false
leak report
http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/lsan/lit_tests/TestCases/use_tls_dynamic.cc?view=markup&pathrev=200748

MemorySanitizer (aka MSAN) is a detector of uninitialized reads,
available in Clang (starting from 3.4).
MSAN maintains the shadow bit for every bit of the application memory.
If the shadow bit is set, it indicates that the corresponding bit in
the application memory is poisoned (not initialized). When a thread
(or dynamic TLS) is destroyed, the shadow for the stack (or dynamic
TLS) should be unpoisoned. Otherwise this memory may be reused by a
following mmap call and MSAN will report false positives.
Example: test case where not knowing about dynamic TLS leads to false
uninitialized memory report
http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/msan/lit_tests/dtls_test.c?view=markup&pathrev=199980
</details>


======================= Boundaries of stack and static TLS
=======================

Proposed interface for retrieving boundaries of stack and static TLS:

// Returns the stack bounds for the current thread.
// For the main thread both values are assigned to __libc_stack_end.
// This function does not have any visible side effect (such as
calling an exported function).
void __libc_get_stack_bounds(void **stack_beg, void **stack_end);

// Get the bounds of static tls for the current thread.
// This function does not have any visible side effect (such as
calling an exported function).
void __libc_get_static_tls_bounds(void **stls_beg, void **stls_end);


======================== Dynamic TLS ========================

The tools need to be notified about creation/destruction of dynamic TLS
immediately after creation and immediately before destruction.

The most desirable interface would be:

// This function creates a chunk of dynamic TLS.
// It is AS-safe.
// A tool may interpose this function and either replace of wrap it.
void *__libc_create_dynamic_tls(size_t size, size_t alignment);

// This function destroyes a chunk of dynamic TLS created by
__libc_create_dynamic_tls.
// It is AS-safe.
// A tool may interpose this function and either replace of wrap it.
void __libc_destroy_dynamic_tls(void *dtls, size_t size);

Almost as attractive alternative is to have a way to register a
callback that is called after DTLS creation and before DTLS
destruction.

Less attractive (and likely not vital at all) alternative is to have
an interface to iterate DTLS chunks in the current (or in a given)
thread.

typedef void (*dtls_callback_t)(void *dtls_beg, void *dtls_end, size_t
dso_id, void *arg);

// Iterate over all DTLS chunks in the given thread.
// Is 't != pthread_self()', the thread 't' should be blocked before
and during this call.
// NOTE: the semantics in case 't != pthread_self()' is not clear!
// AS-safe.
__libc_iterate_dynamic_tls(pthread_t t, dtls_callback_t cb, void *arg);

This approach is problematic for multiple reasons:

The tools need to be notified about the DTLS creation/destruction
immediately, so they will need to call __libc_iterate_dynamic_tls from
__tls_get_addr/dlclose interceptors and in thread shutdown code.

This means greater performance penalty due to required __tls_get_addr
interceptor. Besides, the semantics of __libc_iterate_dynamic_tls
called in dlclose from a different thread is unclear.

__libc_iterate_dynamic_tls should be AS-safe because it may be called
from a signal handler while e.g. __tls_get_addr is being called from a
regular code.

================== Callback for thread exit  ==================
Tools such as ASAN, MSAN and LSAN need to perform various cleanup
actions just before a thread exits.

Currently these tools use an ugly hack to get notified about thread's
destruction: call pthread_setspecific recursively
PTHREAD_DESTRUCTOR_ITERATIONS times; when the
PTHREAD_DESTRUCTOR_ITERATIONS-th call is made we consider the thread
as dead. A cleaner interface would be much appreciated (not sure if it
easy to do or at all possible)

// Register callback to be called right before the thread is totally destroyed.
// The callbacks are chained, they are called in the order opposite to
the order they were registered.
// The callbacks must be registered only before any threads were
created, at most 8 callbacks can be registered.
// No signals may arrive during the calls to these callbacks;
immediately after the last of these calls the thread is dead.
void __libc_register_thread_exit_callback(void (*cb)());


================== More info ==================
https://sourceware.org/glibc/wiki/ThreadPropertiesAPI
https://sourceware.org/bugzilla/show_bug.cgi?id=16291

On Tue, Apr 22, 2014 at 11:51 PM, Roland McGrath <roland@hack.frob.com> wrote:
> The meat of the discussion needs to be on the list, not in bugzilla or wiki
> pages.  Just referring to them is not enough.  A proposal should be written
> up as a posting on this list.  It should start with a detailed, high-level
> description of the requirements so we can hash those out and agree on them
> before discussing the actual API.