This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH 0/N] test-suite improvement - PASS/FAIL: initial patch


On Mon, 17 Sep 2012 11:41:36 +0000, Joseph wrote:
> This doesn't readily give a list of passing tests, though you can get
> that by using "make --debug=b -k check".  But really the point is so
> that not everyone needs to reinvent the wheel in such ways - so that
> you can simply use standard glibc testsuite machinery to give a list
> of test results.

Agreed, just that I would to have preferred to see something that can
be built upon to make a standard testsuite of sorts and this approach
of a file per test doesn't seem the kind of thing that would take our
testsuite too far.

If a file per test is the only way out, I would be more comfortable
patching test-skeleton.c to write PASS/FAIL/etc. as the last line of the
testcase.out file. I know that there are a lot of tests that don't use
the skeleton and the right thing to do would be to fix those in the
process.

> Note that eventually you don't just want PASS and FAIL; you should
> have other statuses such as UNRESOLVED and UNSUPPORTED:
> 
> * UNRESOLVED: a test detected a condition (such as an out-of-memory
> error) meaning it couldn't test what it was meant to test.  Also,
> ideally you'd get PASS or FAIL for compiling/linking a test as well
> as running it, and if those failed then the run of the test would be
> UNRESOLVED.
> 
> * UNSUPPORTED: the test depends on some feature not supported in this 
> configuration of glibc or the system on which the test was run.
> (Some tests aren't built at all in particular configurations; it
> would be best to show those as UNSUPPORTED.  Some may just exit early
> if something isn't supported, e.g. tst-setcontext if getcontext
> returns with an ENOSYS error.)
> 
> Actually implementing those may require reviewing how all the
> existing tests use exit statuses so that suitable statuses can be
> reserved to signal UNRESOLVED / UNSUPPORTED.

Agreed, and for this too, test-skeleton.c seems like the best place to
start to me. And for that, the tests that don't use the skeleton need to
be fixed.

These thoughts are not originally mine of course - they have been
derived and built upon an earlier thread ;)

http://sourceware.org/ml/libc-alpha/2012-03/msg01084.html

So to conclude, my point is that we could use a quick hack like this to
get the PASS/FAIL result now (which could be gotten with other hacks I
mentioned earlier), but the real problem of the tests in our testsuite
not being standardized, remains.

Regards,
Siddhesh


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]