This is the mail archive of the mailing list for the glibc project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Summary for the glibc benchmark BoF

On 08/18/2015 01:10 PM, Steven Munroe wrote:

I don't think IBM is saying that the current benchmarks are complete or
completely representative. We are saying that current benchmarks are
what we have, and a assertion that the lack of some hypothetical
"better" benchmark, should not be used as an excuse to block a patch.
That seems reasonable to me.

Building a benchmarking suite is an iterative process -- you start with what you can and build around it. Over time some tests may be superceded by more representative tests, but until those more representative tests exist, you live with what you've got.

I would personally like to see more representative benchmarks based on
actual usage.
I can't think anyone would say that more representative benchmarks would be a bad idea.

I would also assert that benchmarks should be split into representative
(of normal usage) and extreme (for example testing for quadratic
behavior or only testing for the needle match at the very end of the
haystack) categories. And clearly labeled as such.
Agreed 100%. Creating reasonable buckets for tests is valuable. And for a performance suite, I can certainly see the value in clearly marking normal vs extreme/corner case tests.

We've done it in GCC and it's proven helpful. We're testing for different things, but the fundamental concepts are the same. If test XYZ fails, the quicker you can parse the impact the better. It's lame when the developer has to either "just know" or read the source to find out that the test is some corner case condition.

We have buckets for things like ieee-fp conformance, testing language limits, extension conformance, quality of debugging output, thread safety, etc etc.

The one thing we didn't do was go back to the old tests and put them into the new buckets, though we do try and route new tests into the right buckets.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]