Re: [2.20] [4/6] Enumerate tests with special rules in tests-special variable

[Brooks Moses <bmoses at google dot com>]

On Mon, Feb 10, 2014 at 3:55 PM, Joseph S. Myers
<joseph@codesourcery.com> wrote:
> On Mon, 10 Feb 2014, Brooks Moses wrote:
>> How would this interact with the idea you mentioned in an earlier patch in the
>> series of breaking up tests that have an "examine the output" step into
>> multiple tests?  Do we end up having multiple lines in the enumeration for
>> them?
>
> Yes.  (Sometimes the initial output generation might not need any special
> makefile rules at all.)

Sounds good to me, then.  And you mentioned there are other tests with
interdependencies anyway, so this adds nothing new.

>> > * Should tests that don't generate output files be changed (in a
>> >    separate patch or patches) to do so?
>>
>> What would the output files contain?
>
> Whatever, possibly empty, goes on standard output at present.  For
> example, for the check-abi tests, the diffs compared to the ABI baselines
> (nonempty only if the test fails).  That would have the advantage that
> "make check", in a directory where some tests have already been run,
> doesn't rerun those tests as it does at present, unless the dependencies
> indicate they need to be rerun.

Makes sense.  I suppose there's no harm in the file being empty.

>> If there are runtime errors like a segfault, will they get captured in the
>> output file?  If so, I think that's a strong argument for always having an
>> output file.
>
> Segfaults will get reported by the shell used by make, so not captured.
> The present practice is that only stdout not stderr gets redirected (this
> is questionable, as is tests using both anyway without a clear
> distinction) - but the reporting of a segfault happens after the
> segfauling process has exited, meaning output redirection for that process
> has no effect on the report of the segfault.

Fair point -- I suppose a segfault is a bad example.  (You could
capture it by appending a line for a nonzero exit code onto the .out
file, though, to pull in that idea from my comments on another patch.)

On the other hand, redirecting stderr to the .out file (or a .err file
if we really want to keep the streams distinct) seems like a very good
idea to me, and this _would_ capture things like asserts failing
inside glibc itself.  Given that, I would say that always having a
.out file is a good thing because there is always a possibility that
it will contain a meaningful error message, even when it's normally
empty.  In my experience, things that get printed to the terminal
window during a test run might as well be going to /dev/null, and
capturing everything is the right answer.

- Brooks