Re: [PATCH RFC] introduce dl_iterate_phdr_parallel

[Gleb Natapov <gleb at scylladb dot com>]

On Mon, Aug 01, 2016 at 04:26:20PM -0400, Carlos O'Donell wrote:
> On 07/25/2016 10:23 AM, Gleb Natapov wrote:
> > Problem: exception handling is not scalable. Attached program mt.cc
> > demonstrates this easily. It runs around 1 seconds with 1 thread, but
> > with only 4 threads it takes 4.5 seconds to complete. The reason is
> > locks that are taken on unwind path.
> > 
> > There are two locks right now.
> > 
> > Fist is in libgcc's _Unwind_Find_registered_FDE and it is eliminated by
> > https://gcc.gnu.org/ml/gcc-patches/2016-07/msg01629.html (for dynamic
> > executables only, but this is common case).
> > 
> > Second one is dl_load_write_lock in __dl_iterate_phdr. It serves dual
> > purpose: it stops the list of loaded objects from been modified while
> > iterating over it and it makes sure that more than one callback will
> > not run in parallel. This means that even if we will find a cleaver way
> > to make __dl_iterate_phdr iterate over the objects list locklessly the
> > lock will still have to be taken around the call to callback to preserve
> > second guaranty.
> 
> This is an X Y problem.
> 
> I don't think the solution is to add a new dl_iterate_phdr_parallel interface,
> but instead to rethink why C++ exceptions need to dl_iterate_phdr at all.
> 
> Exactly what information does C++ need and why?
>  
It needs fdes for all functions in the current stack to see which one
can handle current exception. The dl_iterate_phdr callback takes a pc
as a parameter and goes over PT_LOAD sections and checks that given
address falls into one of them, it also finds eh_frame segment in the
same loop. If the pc belongs to the current object it iterates over all
its fdes and finds one responsible for the pc and returns it.

> We have been discussing this internally at Red Hat, but I'd like to get your
> external opinion on the situation.
> 
We are developing Seastar framework [1] and unscalablilty of C++ exception
handling hurts us a lot.

This is asynchronous framework where application code never sleeps
outside of event loop.  In the framework we are going into great length
to keep cpus as isolated from one another as possible and minimize locked
memory access, we even reimplemented std::string and std::shared_ptr to
eliminate locking operations. We also make an afford to enter kernel as
little as possible (and if we do we call functions that do not sleep). As
a result if userspace networking stack is used you never see any kernel
functions in perf profile.

All this breaks down when errors start happening. Isolated exception
here and there are not really noticeable, but sometimes many errors happen
simultaneously and on several cpus (networking disconnects and all
request waiting on it throw exception or disk is overloaded and request
start to timeout). When the exception storm is happening the system
stops functioning since locking put threads to sleep outside of even
loop and it prevents them from doing any work until exception is handled.

The problem is so serious that we consider shipping patched libgcc and
glibc with our application, but this should be only temporary solution
until the patches (or some variation of them) are unstreamed.

[1] http://www.seastar-project.org/

--
			Gleb.