This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
[PING] Wordexp benchtest: good, bad and unreliable.
- From: OndÅej BÃlka <neleai at seznam dot cz>
- To: Carlos O'Donell <carlos at redhat dot com>
- Cc: libc-alpha at sourceware dot org
- Date: Tue, 16 Jun 2015 19:32:55 +0200
- Subject: [PING] Wordexp benchtest: good, bad and unreliable.
- Authentication-results: sourceware.org; auth=none
- References: <20150513124945 dot GA4067 at domone> <55538A90 dot 9000908 at redhat dot com> <20150513183554 dot GA31157 at domone> <55539E20 dot 4010506 at redhat dot com> <20150606105711 dot GA7215 at domone>
So Carlos whats your comment on benchmark results?
On Sat, Jun 06, 2015 at 12:57:11PM +0200, OndÅej BÃlka wrote:
> On Wed, May 13, 2015 at 02:55:28PM -0400, Carlos O'Donell wrote:
> > On 05/13/2015 02:35 PM, OndÅej BÃlka wrote:
> > > On Wed, May 13, 2015 at 01:32:00PM -0400, Carlos O'Donell wrote:
> > >> On 05/13/2015 08:49 AM, OndÅej BÃlka wrote:
> > >>> Hi, as I read Paul's wordexp patch I found that you use inefficient
> > >>> idiom. Checking character membership with strchr is slow due to startup
> > >>> cost. Much better is just use table lookup.
> > >>
> > >> How did you test it was faster?
> > >>
> > > You don't need to look at barometer to see if its raining. It does
> > > single memory access which takes around 5-6 cycles is faster than strchr that takes ~30 cycles
> > > and cannot be faster as it also needs to do a memory access.
> > >
> > >
> > >> Could you please add a wordexp microbenchmark to show the gains?
> > >>
> > > I could. Its easy to make microbenchmark that shows improved performance.
> > > Also its easy to make microbenchmark that makes it a regression. Just
> > > use 128 byte IFS env. variable and call wordexp("x", foo, 0)
> > > I could also make microbenchmark that is inconclusive as performance
> > > difference is lost in syscall overhead of finding filenames by glob.
> >
> > And those would all represent various workloads which we care about?
> >
> > I'm fine having 3 different wordexp microbenchmarks.
> >
> > > So I wont make microbenchmark as it would be useless exercise for pretty
> > > obvious case.
> >
> > OK.
> >
> > > You would need runtime profiling to get something useful.
> >
> > I don't agree with that. I think a microbenchmark of wordexp would be useful
> > as a regression test for this case.
> >
> > Cheers,
> > Carlos.
> >
>
> As I mentioned before for wordexp performance depends on too many
> parameters. So here is microbenchmark that shows some improvements and
> some regressions. There is lot of noise. I ran these three times and on
> one * and ffffffffoo pattern become twice slower for no reason.
>
> So how do you decide with conflicted data like this?
>
>
> old implementation
> wordexp
> Pattern foo flags 0: 1081.77
> Pattern ffoo flags 0: 621.516
> Pattern ffffoo flags 0: 697.562
> Pattern ffffffffoo flags 0: 867.922
> Pattern ffffffffffffffffoo flags 0: 1110.75
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1567.92
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2541.03
> Pattern * flags 0: 141070
> Pattern foo flags 0: 1061.08
> Pattern ffoo flags 0: 1190.48
> Pattern ffffoo flags 0: 1146.83
> Pattern ffffffffoo flags 0: 1407.77
> Pattern ffffffffffffffffoo flags 0: 1734.33
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2143.02
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3100.41
> Pattern * flags 0: 205510
>
>
> wordexp
> Pattern foo flags 0: 1081.84
> Pattern ffoo flags 0: 672.422
> Pattern ffffoo flags 0: 704.828
> Pattern ffffffffoo flags 0: 875.188
> Pattern ffffffffffffffffoo flags 0: 1089.7
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1594.03
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2575.28
> Pattern * flags 0: 144160
> Pattern foo flags 0: 1062.45
> Pattern ffoo flags 0: 1099.72
> Pattern ffffoo flags 0: 1153.95
> Pattern ffffffffoo flags 0: 1391.45
> Pattern ffffffffffffffffoo flags 0: 1820.36
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2129
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3171.5
> Pattern * flags 0: 208475
>
>
> wordexp
> Pattern foo flags 0: 1086.2
> Pattern ffoo flags 0: 615.078
> Pattern ffffoo flags 0: 688.594
> Pattern ffffffffoo flags 0: 851.984
> Pattern ffffffffffffffffoo flags 0: 1085.48
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1562.95
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2540.08
> Pattern * flags 0: 287636
> Pattern foo flags 0: 1068.11
> Pattern ffoo flags 0: 1092.5
> Pattern ffffoo flags 0: 1143.7
> Pattern ffffffffoo flags 0: 1380.66
> Pattern ffffffffffffffffoo flags 0: 1763.88
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2165.72
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3095.41
> Pattern * flags 0: 207373
>
> new
>
>
> wordexp
> Pattern foo flags 0: 1161.95
> Pattern ffoo flags 0: 631.016
> Pattern ffffoo flags 0: 714.797
> Pattern ffffffffoo flags 0: 2209.84
> Pattern ffffffffffffffffoo flags 0: 1024.17
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1408.22
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2223.39
> Pattern * flags 0: 146925
> Pattern foo flags 0: 1660.66
> Pattern ffoo flags 0: 1673.34
> Pattern ffffoo flags 0: 1730.23
> Pattern ffffffffoo flags 0: 2030.42
> Pattern ffffffffffffffffoo flags 0: 2245.59
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2512.95
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3312.95
> Pattern * flags 0: 210492
>
>
> wordexp
> Pattern foo flags 0: 1184.19
> Pattern ffoo flags 0: 634.062
> Pattern ffffoo flags 0: 683.688
> Pattern ffffffffoo flags 0: 826.891
> Pattern ffffffffffffffffoo flags 0: 1020.27
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1405.89
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2185.44
> Pattern * flags 0: 140257
> Pattern foo flags 0: 1649.77
> Pattern ffoo flags 0: 1668.39
> Pattern ffffoo flags 0: 1722.23
> Pattern ffffffffoo flags 0: 1924.27
> Pattern ffffffffffffffffoo flags 0: 2322.56
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2502.8
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3306.39
> Pattern * flags 0: 208003
>
>
> wordexp
> Pattern foo flags 0: 1097.7
> Pattern ffoo flags 0: 655.891
> Pattern ffffoo flags 0: 692.453
> Pattern ffffffffoo flags 0: 860.203
> Pattern ffffffffffffffffoo flags 0: 1009.38
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 1426.02
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 2179.64
> Pattern * flags 0: 144492
> Pattern foo flags 0: 1668.47
> Pattern ffoo flags 0: 1684.38
> Pattern ffffoo flags 0: 1718.75
> Pattern ffffffffoo flags 0: 1932.91
> Pattern ffffffffffffffffoo flags 0: 2294.95
> Pattern ffffffffffffffffffffffffffffffffoo flags 0: 2514.89
> Pattern ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo flags 0: 3329.14
> Pattern * flags 0: 211152
>
> diff --git a/benchtests/Makefile b/benchtests/Makefile
> index 8e615e5..fb737da 100644
> --- a/benchtests/Makefile
> +++ b/benchtests/Makefile
> @@ -35,7 +35,7 @@ string-bench := bcopy bzero memccpy memchr memcmp memcpy memmem memmove \
> strcat strchr strchrnul strcmp strcpy strcspn strlen \
> strncasecmp strncat strncmp strncpy strnlen strpbrk strrchr \
> strspn strstr strcpy_chk stpcpy_chk memrchr strsep strtok \
> - strcoll
> + strcoll wordexp
> string-bench-all := $(string-bench)
>
> # We have to generate locales
> diff --git a/benchtests/bench-wordexp.c b/benchtests/bench-wordexp.c
> new file mode 100644
> index 0000000..ffadbcc
> --- /dev/null
> +++ b/benchtests/bench-wordexp.c
> @@ -0,0 +1,97 @@
> +/* Measure wordexp functions.
> + Copyright (C) 2015 Free Software Foundation, Inc.
> + This file is part of the GNU C Library.
> +
> + The GNU C Library is free software; you can redistribute it and/or
> + modify it under the terms of the GNU Lesser General Public
> + License as published by the Free Software Foundation; either
> + version 2.1 of the License, or (at your option) any later version.
> +
> + The GNU C Library is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + Lesser General Public License for more details.
> +
> + You should have received a copy of the GNU Lesser General Public
> + License along with the GNU C Library; if not, see
> + <http://www.gnu.org/licenses/>. */
> +
> +#define TEST_MAIN
> +#define TEST_NAME "wordexp"
> +#include "bench-string.h"
> +
> +#include <wordexp.h>
> +
> +typedef int (*proto_t) (const char *, wordexp_t *, int);
> +
> +IMPL (wordexp, 1)
> +
> +
> +static void
> +do_one_test (impl_t *impl, const char *s1, int flag)
> +{
> + size_t i, iters = INNER_LOOP_ITERS;
> + timing_t start, stop, cur;
> +
> + TIMING_NOW (start);
> + for (i = 0; i < iters; ++i)
> + {
> + wordexp_t out;
> + CALL (impl, s1, &out, flag);
> + }
> + TIMING_NOW (stop);
> +
> + TIMING_DIFF (cur, start, stop);
> +
> + TIMING_PRINT_MEAN ((double) cur, (double) iters);
> +}
> +
> +
> +static void
> +do_test (const char *s1, int flag)
> +{
> +
> + printf ("Pattern %s flags %i:", s1, flag);
> +
> + FOR_EACH_IMPL (impl, 0)
> + do_one_test (impl, s1, flag);
> +
> + putchar ('\n');
> +}
> +
> +static int
> +test_main (void)
> +{
> + test_init ();
> +
> + printf ("%23s", "");
> + FOR_EACH_IMPL (impl, 0)
> + printf ("\t%s", impl->name);
> + putchar ('\n');
> +
> + do_test ("foo", 0);
> + do_test ("ffoo", 0);
> + do_test ("ffffoo", 0);
> + do_test ("ffffffffoo", 0);
> + do_test ("ffffffffffffffffoo", 0);
> + do_test ("ffffffffffffffffffffffffffffffffoo", 0);
> +do_test ("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo", 0);
> +
> +
> + do_test ("*", 0);
> + setenv("IFS"," \
> + ", 1);
> +
> + do_test ("foo", 0);
> + do_test ("ffoo", 0);
> + do_test ("ffffoo", 0);
> + do_test ("ffffffffoo", 0);
> + do_test ("ffffffffffffffffoo", 0);
> + do_test ("ffffffffffffffffffffffffffffffffoo", 0);
> +do_test ("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffoo", 0);
> + do_test ("*", 0);
> +
> + return ret;
> +}
> +
> +#include "../test-skeleton.c"
--
Increased sunspot activity.