Re: [PATCH] randomize benchtests

[OndÅej BÃlka <neleai at seznam dot cz>]

On Mon, Apr 22, 2013 at 05:05:13PM +0200, chrubis@suse.cz wrote:
> Hi!
> > > You can use 'cpufreq-info -f' on Linux, on BSD's you need to play with
> > > sysctl. There is a sysctl 'emulation' on Linux that uses /proc, but that
> > > doesn't work for CPU speed.
> > > 
> > It looks that reading file is saner than exec followed by reading
> > output.
> 
> The actual code is pretty simple:
> 
> int speed;
> 
> FILE *f = popen("cpufreq-info -f", "r");
> 
> if (f == NULL)
> 	//ERROR
> 
> if (fscanf(f, "%i", &speed) != 1)
> 	//ERROR
> 
> pclose(f);
>
Actually we are interested at max speed, benchmark will spend most time
(insert nonsensial loop) there.

Code above will give wrong result when cpufreq-info gets scheduled
to other processor.

 FILE *f = popen("cpufreq-info -l", "r");
 if (fscanf(f, "%i %i",&_ ,&speed) != 1)

> > > But be wary that CPU speed is subject to change when CPU scaling is
> > > active, which is on on most of the modern systems.
> > > 
> > This with other factors is reason why we should test all implementations
> > together and select implementations randomly. They will be affected by
> > same amount so their speed differences will not be affected. 
> > 
> > When we test them sequentialy overhead will be different each time
> > making comparing them almost worthless.
> 
> Not sure if I follow you correctly. What I wanted to say is that the
> actuall CPU frequency changes depending on the system load. If you read
> it before you start the benchmarks, it will likely be incorrect as the
> system was possibly idle and the benchmarks made the cpu freq governor
> to switch to higher speeds as the system load increased. So converting
> CPU cycles to time and back is meaningless unless you fixed the CPU
> speed to certain value manually. 

It was separate comment from that. 

We also need to set cpu affinity to avoid cpu migrations.

Ondra