Re: [RFC] Lock elision "test suite", first draft

[Dominik Vogt <vogt at linux dot vnet dot ibm dot com>]

Test results on z/Architecture (zEC12)
--------------------------------------

Test Setup:

  An lpar with eight CPUs on a ZEC12.  8 GiB RAM.

Test execution:

  Each test case was run ten times in a loop using an unpatched
  glibc-2.15 (Ubuntu standard) ("old glibc"), a new glibc-2.18
  with elision disabled ("elision n/a"), and a new glibc-2.18
  with elision enabled ("elision on").

  The elision patches from the thread "[RFC][PATCH 0/2] Port of
  lock elision on z/Architecture" have been used.  Glibc has been
  compiled with a recent development version of gcc that
  implements the htm builtins (__builtin_tbegin etc.).

Test evaluation:

  The best result out of the ten is used for further calculations.
  The result for glibc-2.15 is considered to be 100%.  The best
  result for each setup is compared to the result with glibc-2.15.
  The relative performance is listed as "pct_best".

  The values pct_mindev, pct_maxdev and pct_stddev give an
  indication of the variation of the individual ten test results.
  pct_mindev and pct_maxdev denote the relative deviation of the
  minimum and maximum value of the ten compared with the average
  over all ten.  I.e. pct_mindev 50% means that the lowest
  measured value is 50% of the average.

  pct_stddev denotes the relative standard deviation of the ten
  values, i.e. it's (100 * stdandard-deviation / average-value).


./ets-t1a
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 32.60...%
pct_maxdev: 3.65...%
pct_stddev: 10.86...%

elision n/a:
------------
pct_best  : 102.90...%
pct_mindev: 24.26...%
pct_maxdev: 2.99...%
pct_stddev: 8.09...%

elision on:
-----------
pct_best 51.92...%
pct_mindev: 17.54...%
pct_maxdev: 2.63...%
pct_stddev: 5.87...%

./ets-t1b
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 1.24...%
pct_maxdev: 0.23...%
pct_stddev: 0.42...%

elision n/a:
------------
pct_best  : 104.18...%
pct_mindev: 2.70...%
pct_maxdev: 0.90...%
pct_stddev: 1.16...%

elision on:
-----------
pct_best 49.39...%
pct_mindev: 0.45...%
pct_maxdev: 0.74...%
pct_stddev: 0.42...%

./ets-t2
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 50.95...%
pct_maxdev: 40.79...%
pct_stddev: 27.84...%

elision n/a:
------------
pct_best  : 93.56...%
pct_mindev: 37.49...%
pct_maxdev: 26.86...%
pct_stddev: 19.68...%

elision on:
-----------
pct_best 9.26...%
pct_mindev: 7.09...%
pct_maxdev: 6.82...%
pct_stddev: 4.50...%

./ets-t3
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 4.57...%
pct_maxdev: 5.43...%
pct_stddev: 3.61...%

elision n/a:
------------
pct_best  : 96.79...%
pct_mindev: 9.82...%
pct_maxdev: 9.42...%
pct_stddev: 5.71...%

elision on:
-----------
pct_best 593.48...%
pct_mindev: 5.73...%
pct_maxdev: 4.09...%
pct_stddev: 3.43...%

./ets-t4a
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 2.44...%
pct_maxdev: 1.58...%
pct_stddev: 1.39...%

elision n/a:
------------
pct_best  : 102.30...%
pct_mindev: 16.39...%
pct_maxdev: 3.12...%
pct_stddev: 5.58...%

elision on:
-----------
pct_best 31.69...%
pct_mindev: 9.53...%
pct_maxdev: 21.99...%
pct_stddev: 13.36...%

./ets-t4b
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 0.27...%
pct_maxdev: 0.19...%
pct_stddev: 0.12...%

elision n/a:
------------
pct_best  : 100.16...%
pct_mindev: 0.12...%
pct_maxdev: 0.10...%
pct_stddev: 0.06...%

elision on:
-----------
pct_best 56.83...%
pct_mindev: 9.43...%
pct_maxdev: 11.37...%
pct_stddev: 8.62...%

./ets-t4c
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 3.80...%
pct_maxdev: 2.82...%
pct_stddev: 2.45...%

elision n/a:
------------
pct_best  : 106.27...%
pct_mindev: 6.46...%
pct_maxdev: 12.54...%
pct_stddev: 5.03...%

elision on:
-----------
pct_best 56.44...%
pct_mindev: 13.54...%
pct_maxdev: 6.16...%
pct_stddev: 7.36...%

./ets-t5
=========

old glibc:
----------
pct_best  : 100.00...%
pct_mindev: 2.54...%
pct_maxdev: 3.50...%
pct_stddev: 2.14...%

elision n/a:
------------
pct_best  : 103.92...%
pct_mindev: 16.39...%
pct_maxdev: 9.48...%
pct_stddev: 8.68...%

elision on:
-----------
pct_best 48.92...%
pct_mindev: 1.16...%
pct_maxdev: 0.97...%
pct_stddev: 0.55...%

Ciao

Dominik ^_^  ^_^

-- 

Dominik Vogt
IBM Germany