Re: [RFC] Keep updated glibc-bench in openbencharking

[Siddhesh Poyarekar <siddhesh at gotplt dot org>]

On Wednesday 18 October 2017 10:37 PM, Rodriguez Bahena, Victor wrote:
> Lately I have been working on keep the external libc-bench test updated
> with latest version of glibc. The approach that I follow was to get the
> latest version , compile the tests and give the user the possibility to
> run:  
> 
> 1: bench-math
> 2: bench-pthread
> 3: bench-string
> 
> Each one represents the major fields of glibc features to be benchmarking
> ( as far as I can see on benchmark/Makefile ). In order to run these
> subgroups of tests I send a patch: "Group benchtests in
> math/pthread/string² The complete patch for the operbenchmarking framework

I reviewed this patch earlier today.  To summarize, it is a good idea to
have these divisions, but I'd like to see it done in a slightly
different way so that it can be extensible.  Detailed comments are in
the patch review.

> is here : 
> https://github.com/VictorRodriguez/phoronix-tests/commit/cf28f6399ed8aed08c
> 15a8c6c8717bcc22a1f39a
> 
> It has already been sent to open benchmarking for testing.
> 
> Few points to check:
> 
> 1) Is it ok just to test these subgroups ? Do we need more ?

Among the existing subgroups, there is also the malloc group of tests to
test the malloc implementation.

These subgroups were added over time on a need basis.  For example, most
of the string benchmarks were just simple dummy tests that test L1
performance of the string routines.  We are improving them continuously,
adding new benchmarks as we see fit, trying to get in more
representative workloads.

As far as glibc as a whole is concerned, there are a lot of gaps in our
performance testing and they will hopefully be addressed as interested
people contribute tests they want to see improved.

> 2) in open benchmarking patch is remove the line :
> 
> $(PYTHON) scripts/validate_benchout.py $(objpfx)bench.out
> \		scripts/benchout.schema.json
> 
> 
> Due to the fact that it was failing with these output:
> 
> Invalid benchmark output: 'workload-spec2006.wrf' does not match any of
> the regexes: '^[_a-zA-Z0-9]*$¹ or
> Invalid benchmark output: Additional properties are not allowed
> ('workload-spec2006.wrf' was unexpected)

Interesting, that may be a bug in the validation script, which does not
expect the hyphen and the dot.

> 3) Any feedback is more than welcome, im happy to keep the external test
> in sync with glibc latest release and help others to measure the
> performance of glibc versions

Thanks for doing this.  Hopefully feedback from openbenchmarking will
help us improve our benchmark inputs and programs.

Siddhesh