Re: performance of multithreading gets gradually worse under gdb

[Michael Snyder <msnyder at vmware dot com>]

Markus Alber wrote:
>  Thank you for the quick response.

You're welcome, but let's keep the discussion on the public list, so 
that other maintainers may jump in if so inclined.


>  It allocates about 100kB per iteration.

Hmmm, and that's for roughly 36 thread start/stops, so it could be
losing roughly 3000 bytes per thread.  That's much bigger than my
first guess would have been (a "struct thread_info" is only 336 bytes).

>  One interesting finding might also be:
>
>  I terminated the process when an iteration took about 3 min (instead of 
>  1 sec)
>  and gdb had about 115MB allocated.

I assume that at this point, your system still had plenty of ram to 
spare?  It wasn't simply swapping?

>  On starting the application again, it ran alright for a while and the 
>  gdb memory allocation
>  stayed constant. When it finally started to grow again, the application 
>  slowed down, and became
>  slower with every iteration - the usual picture.
>
>  I attached a sample file from the application where the computation 
>  bifurcates into
>  the worker threads. This is one of three instances per iteration, but 
>  they all follow the
>  same pattern.

I was really hoping for a stripped-down sample that we could compile and 
run.


> The machine has 2x6 cores x 3 instances per iteration = 
>  36 worker threads per iteration.

x86 architecture?

>  On another note, I tried to compile gdb-6.5 on my machine (because it 
>  was the release I
>  used to work with before, without problems) and configure comes back 
>  with an error that it cannot find a termcap lib. There is none on the 
>  SuSE. Which package would I need to install?

That would be libncurses, I think.


>  On Wed, 02 Feb 2011 12:27:58 -0800, Michael Snyder wrote:
> > Markus Alber wrote:
> > >  Hello,
> > > I have experienced the following problem:
> > > I'm debugging a number-crunching application which spawns a lot (36) 
> > > little
> > >  worker threads per iteration. The system does typically OoM 200 
> > > iterations.
> > >  Although each of them should take about the same amount of time, 
> > > the performance
> > >  gets worse with every iteration and becomes excruciatingly slow.
> > > A system monitor reveals that gdb allocates more memory with every 
> > > iteration,
> > >  i.e. with every 36 threads started and finished. The CPU load of 
> > > GDB goes up, too.
> > >  The CPU usage of the application goes down. Compared to the solo 
> > > performance, it
> > >  gets slower by a factor 20 and more, if run long enough.
> > > The application behaves perfectly when run by itself. The 
> > > multi-threaded part is not
> > >  debugging compiled when this behaviour occurs.
> > > The distribution is SuSE 11.3 / gdb 7.1.
> > > Is there anything I can change about this behaviour, any options of 
> > > gdb that need to
> > >  be set in these circumstances?
> > Interesting.
> >
> > By how much does gdb's memory allocation increase?
> > In total or, if possible, per iteration?  This might
> > give is a clue as to where to look.
> >
> > Do you think you could write a simple sample program that
> > allocates threads in a manner similar to your application?
> >
> > Thanks,