Re: Smoke test for glibc

[Andreas Jaeger <aj at suse dot com>]

On Monday, October 22, 2012 09:28:11 Rich Felker wrote:
> On Mon, Oct 22, 2012 at 09:23:13AM -0400, Carlos O'Donell wrote:
> > On Mon, Oct 22, 2012 at 9:00 AM, Andrey Ponomarenko
> > 
> > <aponomarenko@rosalab.ru> wrote:
> > > I've recently written an article [1] about smoke testing of Glibc
> > > API using a modern approach. This kind of test is needed to
> > > verify that most of the library functions pass a simple test in
> > > the particular environment, i.e. don't crash, don't emit any
> > > signal, don't hang and don't exit with non-zero error code. The
> > > test generator takes only a path to the installed glibc tree in
> > > order to compose input arguments for all functions (reasonable in
> > > most, but unfortunately not all, cases) and then create, build
> > > and run the test cases.
> > > 
> > > I've added a link to this article on the Release Process page
> > > (section 5.6. Testing), so that glibc maintainers can run this
> > > additional test when preparing the release and look at the
> > > behaviour of all functions under the smoke test. The sample test
> > > report for 2.16 looks like this one [2].> 
> > Andrey,
> > 
> > Thanks! This looks very interesting.
> > 
> > Will you continue to work on getting these results clean for glibc?
> > 
> > Going through 154 failed testcases from the smoke test is not
> > something that I want a release manager to be doing.
> 
> I did a quick look through them, and a lot come from the test
> generator not knowing how to satisfy the interface contracts. For
> instance, it's passing {0,1,2,3,4,5} to longjmp and then getting
> surprised when that causes a crash.
> 
> Further, it's passing uninitialized memory to at least some functions,
> which can't be meaningful unless they only use the pointer for
> output. For example, the hang in pthread_spin_lock is simply due to
> passing a pointer to an indeterminate value.


Or opens a directory that does not exists and then uses the file 
descriptor to read data - which gives a SEGV since the file descriptor is 
invalid...

 
> In order to be meaningful, this sort of testing can only be applied to
> functions for which the entire input domain is valid (i.e. mainly
> mathematical functions; certainly nothing that takes a pointer), or
> it must be enhanced to understand the interface contracts and satisfy
> them.

Exactly. Right now it shows too many problems and figuring out which are 
real glibc problems is like looking for the needle in the haystack - so 
far I only found usage problems but no glibc ones.

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg)
    GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126