Re: [PATCH] benchtests/Makefile: Run the string benchmarks four times by default.

[OndÅej BÃlka <neleai at seznam dot cz>]

On Thu, Sep 05, 2013 at 08:51:53AM +0100, Will Newton wrote:
> The intention of my patch - which I may have not made completely clear
> in the commit message - is to improve test stability. What I mean by
> this is that with a physically indexed cache the physical pages
> allocated to the test can have a significant effect on the performance
> at large (e.g. cache size / ways and above) buffer sizes and this will
> cause variation when running the same test multiple times. My aim is
> to average out these differences as it is hard to control for them
> without understanding the details of the cache subsystem of the system
> you are running on.
>
This can be just explained just by having more data. Simply multiplying 
iteration count by four would then do same job. 

Please run your patch ten times and calculate variance. Compare that to
variance when iteration count is increased 4 times and show if there is
improvement. 


 
> Your test appears to be addressing concerns of test validity by
> running a wider range of buffer alignments, which is an important but
> separate concern IMO.
> 
No, your patch will pick src pointer at 4 different physical pages (one
allocated in each run) and calculate average performance.

Mine will pick src pointers in 2000000/4096 = 488 different pages and
calculate average. 

As this makes your patch redundant you will  add unnecessary complexity 
to glibc, there is a very good reason to avoid complexity in a project
like glibc, which is already extremely complex.  Some poor soul will 
need to maintain all of the complicated solutions long into the future.