Re: [PATCH RFC] introduce dl_iterate_phdr_parallel

[Torvald Riegel <triegel at redhat dot com>]

On Mon, 2016-08-01 at 21:49 +0300, Gleb Natapov wrote:
> On Mon, Aug 01, 2016 at 03:06:32PM -0300, Adhemerval Zanella wrote:
> > 2. Another option is to push to cleanup dl_iterate_phdr interface to *not*
> >    require callback serialization and use a rdlock while accessing the
> >    maps list.  With current rwlock implementation performance won't change
> >    as you noticed, however since we are reworking and changing to a more
> >    scalable one [1] the read only pass should *much* better: the 
> >    __pthread_rwlock_rdlock_full should issue just a atomic_fetch_add_acquire
> >    for uncontented read case.
> I saw new rwlock implementation and tested it. It is much better that
> current rwlock, but still almost twice slower than multiple locks.
> Profiling shows that exchange in unlock takes a lot of cpu. No wonder
> since congested locking operation is very expensive. IMO ideal solution
> is array of rwlocks.

The new rwlock is built so that it supports process-shared usage, which
means that we have to put everything into struct pthread_rwlock_t.  This
will lead to contention if you rdlock it frequently from many threads.
There is potential for tuning there because we haven't looked closely at
adding back-off in the CAS loop (and if you tested on an arch without
direct HW support for fetch-add, the CAS loop used instead of that might
also be suboptimal).
Which machine did you test this on?

If we built something custom for this and are willing to make the
wrlock / exclusive-access case much more costly, we can decrease this
overhead.  This could be roughly similar to one lock per thread or a set
of rwlocks as you mentioned, but with less space overhead.