sunday project report, gdb, 2002-11-25

[Michael Elizabeth Chastain <mec at shout dot net>]

After a six-month hiatus, the Sunday Project is back with its bewildering
array of tables!

My test bed is bigger.  I de-coupled the selection of gcc versions and
binutils versions, so that I build and test all versions of gcc with
all versions of binutils.  This helps to identify bugs correlated
with gcc version versus bugs correlated with binutils version.

The test matrix includes gdb 5.2.1, the gdb 5.3 branch, and gdb HEAD,
with 9 versions of gcc (!) and 4 versions of binutils.  My test bed is
still native i686-pc-linux-gnu only.  I'm not going to try to compare
with the last test runs on 2002-05-29, yet.

All of my scripts are available under the GPL if anyone else wants
to set up a test bed or re-use bits of them.  Props to Andrew Cagney
for the original analize [sic] awk script.

Here are some observations:

. I did a full comparison of gdb 5.2 versus gdb 5.2.1 and found no
  significant changes: no regressions and no improvements.

. The "compare by gdb" tables compare gdb 5.2.1 versus the 5.3 branch
  and gdb HEAD.  The following test scripts show regressions from 5.2.1
  to the 5.3 branch:
    gdb.base/condbreak.exp
    gdb.base/ena-dis-br.exp
    gdb.base/printcmds.exp
    gdb.base/selftest.exp
    gdb.c++/annota2.exp
    gdb.c++/local.exp
    gdb.c++/method.exp
    gdb.c++/templates.exp.
  I estimate that 1-2 of these are bugs.  I will investigate this further
  and publish a detailed report.

. gdb/testsuite/gdb.threads/schedlock.exp is very noisy in the gdb 5.3
  branch.  In gdb HEAD, Daniel Jacobowitz has a fix: change the "args"
  array from plain integers to signed integers.  I would like to commit
  this patch to the 5.3 branch also.  With this patch, schedlock.exp
  still has 8 FAILs most of the time on the branch, compared to 2 FAILs
  all of the time on HEAD..

. I still test with gcc 2.95.3.  In the future I will analyze how
  gcc 2.95.3 test results have changed since my last cycle of reports
  (May 2002).

. binutils has been improving.  binutils 2.11.2 caused problems in
  gdb.c++/virtfunc.exp.  binutils 2.12 and binutils 2.12.1 caused
  problems in gdb.base/condbreak.exp, gdb.base/ena-dis-br.exp,
  gdb.base/funcargs.exp, and gdb.c++/ovldbreak.exp.  All of these problems
  are fixed in binutils 2.13 and later versions.

. there are 10-20 places where the test result regress as the gcc version
  increases.  I am looking into these now.  Most of these, but not all,
  are C++ related.

. gcc HEAD has a lot of regressions on c++ stabs+ debugging.  Look at
  the "compare by gcc version" table for gcc=HEAD, binutils=HEAD,
  gformat=stabs+.

. gcc 3.0.4 has a lot of bugs, but there are some tests which PASS on
  gcc 3.0.4 and FAIL on gcc 3.1 and up!  I have to keep gcc 3.0.4 in the
  test bed while I investigate these regressions, but gdb does not have
  to fix any failures with gcc 3.0.4.

. Alas for Jim Blandy and his macro expander, I lost the ability to do
  "-g2" versus "-g3" testing.  I need to shrink my test matrix first and
  then I can add that dimension back in.

My tables are here:

  http://www.shout.net/~mec/sunday/2002-11-25/index.html

Michael C

. Summary

  . Test Matrix

    . Matrix

      target:    native
      host:      i686-pc-linux-gnu%rh-8
      gdb:       5.2.1, gdb_5_3-branch%20021125, HEAD%20021125
      gcc:       2.95.3, 3.0.4, 3.1, 3.1.1, 3.2, 3.2.1, 3.2-7-rh, gcc-3_2-branch%20021125, HEAD%20021125
      binutils:  2.13.90.0.2-2-rh, 2.13.1, binutils-2_13-branch%20021125, HEAD%20021125
      glibc:     2.2.93-5-rh
      gformat:   dwarf-2, stabs+
      count:     198 = *(1 1 3 +(*(8 4) 1) 1 2)

    . Notes

      target and host names are gnu triples with extra information after
      a '%' delimiter.

      versions which start with a number are official releases or snapshots.
      versions which start with a number and end with "-rh" are the
        vendor-supplied versions on my red hat linux host.
      versions named "HEAD" are the cvs HEAD.
      versions with any other name are cvs branches.
      cvs versions show the checkout date after a '%' delimiter.

    . libiberty

      Not tested yet (I need to update my test script).

    . gdb

      . Tables

        . http://www.shout.net/~mec/sunday/2002-11-25/index.html

      . Overview

        gdb 5.2.1:           0 build aborts, 0 test aborts, 512 attention lines
        gdb gdb_5_3-branch:  0 build aborts, 0 test aborts, 361 attention lines
        gdb HEAD:            0 build aborts, 0 test aborts, 348 attention lines

        An attention line is a test with an attention results in one or
        more configurations.

        An attention result is any result except PASS.  This includes
        ERROR, WARNING, FAIL, KPASS, KFAIL, XPASS, XFAIL, UNRESOLVED,
        UNTESTED, UNSUPPORTED, and unknown results.

      . Old bugs fixed

        Not analyzed

      . New bugs detected

        Not analyzed

. Test protocol changes since last report.

  The last report was 2002-05-29, six months ago.
  
  There have been many changes in the tool versions:
    . host: red hat linux 7.2 -> red hat linux 8
    . gdb: gdb 5.2 -> gdb 5.2.1 (full test series, no significant changes found)
    . gdb: gdb_5_2-branch -> gdb_5_3-branch
    . gcc: add gcc 3.1, gcc 3.1.1, gcc 3.2, gcc 3.2.1
    . gcc: gcc-3_1-branch -> gcc-3_2-branch
    . binutils: add 2.13.1, binutils-2_13-branch, HEAD, vendor

  I de-coupled the selection of gcc versions and binutils versions.
  In the past, I built each gcc with just one particular binutils: gcc
  HEAD with binutils HEAD, gcc 2.95.3 with binutils 2.11.2, and so on.
  Now I build the whole cross product of gcc's and binutil's, except for
  the vendor-supplied compiler, which is obviously pre-built with the
  vendor-supplied binutils and suitable for rebuilding.  This makes it
  a lot easier to point to either gcc or binutils when regressions happen.

. Baseline software

  . host i686-pc-linux-gnu%rh-8

    make 3.79.1
    binutils 2.13.1
    gcc 2.95.3
    flex 2.5.4
    bison 1.35
    tcl 8.3.4
    expect 5.33.0
    dejagnu 1.4.2 + FernandoN kfail patch.

    The baseline software is used to build all the other software.
    It is not part of the test matrix.

    I will be upgrading the baseline software as time becomes available,
    starting with dejagnu 1.4.3, which has KFAIL support integrated.

. Analysis

  . libiberty

    . results

      . target native

        . host i686-pc-linux-gnu%rh-8

          No results available yet.

  . gdb

    The last report was 2002-05-29.

      http://www.shout.net/~mec/sunday/2002-05-29/Analysis.txt

    No analysis of current results available yet.