Re: Test suite docs

[Eli Zaretskii <eliz at gnu dot org>]

> Date: Sat, 13 Jan 2007 15:26:07 +0100 (CET)
> From: Mark Kettenis <mark.kettenis@xs4all.nl>
> CC: gdb@sourceware.org
> 
> DejaGnu has an info manual, although it isn't too helpful.

Am I supposed to get it by simply typing "info dejagnu"?  If so,
perhaps the sysadmins didn't install some package, and I should ask
them to.

> But really
> for a normal Unix-like systems, once you've installed DejaGnu and its
> dependencies, running the testsuite is as easy as typing "make
> check-gdb" in the toplevel directory.

I needed the docs to look for answers for the questions I posted, not
for running the suite (which indeed ``just works'' if one types "make
check").  I'm not used to ask questions without first trying to find
the answers in the available docs.

> > But that's just one more reason to have a good user-level
> > documentation in GDB to help overcome these difficulties.
> 
> I wonder if that effort isn't better spent on improving the DejaGnu
> manual.

I will be able to comment on that once I see that manual ;-)

> >   . Where do I find the canonical results for my platform?
> 
> In theory one should not see any FAILS, and one should work on
> eliminating any KFAILS.

Granted, the theory is understood: this is, after all, a test suite,
so all tests should pass.  I was asking about the practice, and
specifically about the current state of affairs.  It'd be great if the
results on each platform where we test regularly would be available
with every snapshot tarball.  Can we perhaps set up the snapshot
script to produce that?

> Apart from the obvious PASS and FAIL, we have:
> 
> XFAIL
> 
>   The test failed but this was expected because of problems out of our
>   control, for example OS or compiler bugs that cannot be easiliy
>   worked around.
> 
> XPASS
> 
>   The test was expected to fail, but passed.  Shouldn't happen, but
>   sometimes we accidentally fix bugs.  It could also be that an OS or
>   compiler bug got fixed, and the testsuite needs to be adjusted to
>   recognize that.
> 
> KFAIL
> 
>   The test was known to fail.  This is not a new bug but a known bug
>   in gdb.
> 
> UNTESTED
> 
>   The system lacks functionality to run the test, for example because
>   of a missing compiler, or an unimplemented feature in the OS or the
>   particular GDB config under test.

Thanks!

> All tests should complete within a reasonable amount of time.  If you
> see any FAILS because of timeouts, there's a reasonable chance it's
> actually the test itself that is broken.

Well, in my case it was the bigcore.exp test, see my other message.

> If running a test on a particular platform really is a bad idea, you
> can add some code to make it bail out.  Many tests in
> testsuite/gdb.arch do this.

Thanks, I will take a look.