Re: [PATCH] support: Introduce new subdirectory for test infrastructure

[Stefan Liebler <stli at linux dot vnet dot ibm dot com>]

On 12/12/2016 03:49 PM, Florian Weimer wrote:
> On 12/12/2016 02:11 PM, Stefan Liebler wrote:
> > Hi Florian,
> >
> > I've observed a new test-failure string/tst-strcoll-overflow after this
> > patch has been commited.
> >
> > The test uses:
> > #define TIMEOUT 300
> > #define EXPECTED_SIGNAL SIGALRM
> > #define EXPECTED_STATUS 0
> >
> > With the old test-skeleton, it either timed out or strcoll returned in
> > time and do_test returns 0. The test passes as WEXITSTATUS (status) ==
> > EXPECTED_STATUS.
> >
> > With the new test-driver, the out-file contains:
> > 0
> > Expected signal 'Alarm clock' from child, got none
>
> My bad, I did not run the xtests. :(
>
> > If do_test would return e.g. 2 and EXPECTED_STATUS is defined to 2, the
> > test would pass. But if the testcase is then called with --direct, it
> > would return 2 instead of 0!
>
> --direct is not compatible with EXPECTED_SIGNAL or EXPECTED_STATUS; the
> caller has to know if any of those expectations apply.  As far as I
> know, that has always been the case.
>
> > Shall the new test-driver be able to handle this case?
>
> I'm not sure what the intended behavior is.  The test says that
> collating such a huge string could take days.  But that's clearly not
> what's happening.  For me, it finishes within 15 seconds.
That's what I see, too.
At least the old test-skeleton does not check if a signal should have 
been delivered if the test returns and EXPECTED_STATUS and 
EXPECTED_SIGNAL is defined.

> Can we assume that the test will run to completion with a 300-second
> timeout (plus TIMEOUTFACTOR scaling)?  Then I'll update the test
> accordingly.
I don't know how long it lasts on other archs.
What happens if the testcase timeouts as the testcase expects a SIGALRM?

> Thanks,
> Florian