Re: Spin barrier

[Ed <icelus2k5 at gmail dot com>]

On 28 August 2010 21:29, Carlos O'Donell <carlos@systemhalted.org> wrote:
> Truthfully I would be more interested on seeing the problems you
> encountered using pthread_barrier_t and what led you to choose a
> custom spin barrier implementation?

The problem with pthread_barrier_t is I believe it uses the OS to
sleep each waiting thread. This means you're dependent on the OS to
bring the threads back. This doesn't happen immediately after the
barrier is triggered. Sometimes it doesn't happen at all for some
threads before a phase is complete, so a thread wakes to immediately
just sync again. This also means the OS is invited to be clever about
where the thread should run, and it interacts badly (pathologically)
with a fast barriers like this. Instead of identifying n threads
running together, with synchronization between them every few hundred
microseconds, the OS (I guess) sees multiple processes each of which
is spending a fair amount of time sleeping. This reasoning is just
conjecture; what you see is the OS pushing the threads onto the same
core, and time slicing between them. It's even worse if you use
pthread_mutex_t for locking as opposed to pthread_spIn_t; the OS will
happily put several threads on the same core and switch between them
at the lock points. You get serial performance (actually quite a bit
worse that serial with the context switching and restart delays).

As for why a custom spin barrier, I wasn't aware of a reliable one
that I could drop into play. There doesn't seem to be any move in
POSIX in this direction. A roll-my-own version using spin locks looked
horrifically inefficient. I know how this one works, and what it's
based on, which is worth a lot if you need the code to definitely
function correctly.

Hope this helps,

Ed