Re: Summary for the glibc benchmark BoF

[Siddhesh Poyarekar <siddhesh at redhat dot com>]

On Tue, Aug 18, 2015 at 02:10:05PM -0500, Steven Munroe wrote:
> I don't think IBM is saying that the current benchmarks are complete or
> completely representative. We are saying that current benchmarks are
> what we have, and a assertion that the lack of some hypothetical
> "better" benchmark, should not be used as an excuse to block a patch.

OK, thanks for clarifying that.  FWIW, I did not mean to imply that
you thought that the benchmarks were complete.  I meant to say that
you think they're not useless or misleading enough that they should be
ripped out altogether.

> I would personally like to see more representative benchmarks based on
> actual usage.

+1.  I've made calls in the past to enhance the inputs or even rewrite
the string benchmarks.  The current benchmarks only measure raw
performance for very specific types of input (i.e. aligned at certain
boundaries, misaligned, etc.)  and don't even try to pretend to
represent any kind of generic workload.  If there are workloads that
matter, contributors should enhance the benchmark to provide that as
an input.

My hope is that full system benchmark workloads would give us that,
but that's as concrete right now as any other claim of representative
workload.

> I would also assert that benchmarks should be split into representative
> (of normal usage) and extreme (for example testing for quadratic
> behavior or only testing for the needle match at the very end of the
> haystack) categories. And clearly labeled as such.

Agreed.

Siddhesh