Re: RFC: libc-testresults mailing list

[Andrew Pinski <pinskia at gmail dot com>]

On Wed, Nov 2, 2016 at 9:48 AM, Joseph Myers <joseph@codesourcery.com> wrote:
> On Wed, 2 Nov 2016, Carlos O'Donell wrote:
>
>> Do we expect that libc-testresults is a superset of those builds we have
>> connected to our buildbot CI infrastructure?
>>
>> Do we expect buildbot test results to go to libc-testresults for archiving?
>
> If we add a script to mail out results then I think it would be
> appropriate to make our buildbot scripts use it (having made sure that
> existing buildbot instances have working outgoing mail and documented that
> as a requirement for buildbot instances).  In that case, yes, it would be
> a superset of the buildbot builds (note Andrew was asking about where to
> send results from a system not hooked up to our buildbot).

Yes; the main reason why I don't have it hooked up to the buildbot is
that I need/want the ability to run other things on the same machine
like even GCC build and test since build/test of glibc only takes an
hour and there is not enough changes going into glibc to dedicate one
machine just for glibc testing and one for gdb testing and one of gcc
testing.

> We can consider questions such as how much goes in test results mails from
> such a script, e.g. do we include the full tests.sum (possibly gzipped and
> attached) or just results that aren't PASS, what standard information
> about the test environment goes in mails, whether to attach (gzipped) .out
> files of failing tests if they aren't too many / too big, how mails should
> declare that they are the standard output of the script to allow for
> automatic parsing.

GCC only sends summary of what fails and what I know, that is usually
enough to track down failures on some target and when it started to
fail and then go back find which exact change broke it.  There are
many machines available on the "GCC" Compile farm to be able to run
tests like this.

>
> It would of course be expected that if someone's build system starts
> flooding the list with multiple mails from identical builds (keeping
> rebuilding the same versions continuously, or mail system problems
> resulting in duplicate mails) then those mails would be blocked until the
> problem is sorted out.

We have had that issue once or twice but nothing in the last 2-3 years
as far as I know.

Thanks,
Andrew

>
> --
> Joseph S. Myers
> joseph@codesourcery.com