Re: libc-alpha Digest 19 Dec 2017 01:07:32 -0000 Issue 6029

[DJ Delorie <dj at redhat dot com>]

"christopher.aoki@oracle.com" <christopher.aoki@oracle.com> writes:
> Just to be clear, by “random benchmarks” do you mean synthetic benchmarks
> based on workloads generated using a pseudo-random number generator, or
> something else?

I meant a benchmark you found on the internet, vs one we've collected as
part of glibc's benchmark collection.

So, for example, if you say "Hey folks, I found this benchmark on
foo.bar.com and it says my patch is wonderful!" then you really haven't
added much to my confidence.  Instead, I'd rather you say "Hey folks, I
ran all the glibc benchmarks and they say my patch is wonderful!" ;-)

Of course, this doesn't mean I don't want to know about benchmarks you
find on the internet!  But if a benchmark is relevent to glibc's goals,
reflects glibc's users and their use cases, and is redistributable (or
we can trace it), we should include it in our corpus of benchmarks.  The
goal here is that everyone should have access to the same set of
benchmarks, so that results are reproducible and use cases aren't
neglected.

Specifically, we don't want to do a glibc release and *then* find out
some major application sees worse performance than before, because we
didn't benchmark that application and thus didn't notice that a change
affected it negatively.