Re: racy tests

[Doug Evans <dje at google dot com>]

On Sat, Jul 11, 2015 at 11:54 AM, Pedro Alves <palves@redhat.com> wrote:
> One thing that I'd like is for this to be part of the testsuite
> itself, rather than separate machinery the buildbot uses.  That way,
> everyone benefits from it, and so that we all maintain/evolve it.
> I think this is important, because people are often confused that
> they do a test run before patch, apply patch, run test, and see
> confusing new FAILs their patch can't explain.

No disagreement there.
I would build it on top of what's there now.
[I'd rather build this up in layers, and not have
overly complicated lower layers.]

A next question that arises is maintaining history.
E.g., how does one diff the results of the current run
with the current "gold standard"?

The way I do it here is to have separate files that augment the
XFAIL/KFAIL markers in the test (it's far easier to maintain a few
files than editing each test's .exp file)
but  I'm not sure it scales well.
[E.g., I need to keep separate files for different compilers,
though there is a #include mechanism for common stuff.]

Alternatively,
If a test run could take as input the gdb.sum file from a baseline
run (e.g., from an unpatched trunk) then that could work.
Buildbot could use the previous run, and Joe-Developer
could either use as input a buildbot run's output file
or run the testsuite twice (e.g., with/without the patch-under-test).
[I wouldn't use gdb.sum specifically, I'm just using it here
for illustration's sake.]