[Bug libc/12232] setrlimit interferes with malloc

["bkorb at gnu dot org" <sourceware-bugzilla at sourceware dot org>]

http://sourceware.org/bugzilla/show_bug.cgi?id=12232

--- Comment #10 from Bruce Korb <bkorb at gnu dot org> 2011-01-07 23:24:18 UTC ---
Here is a gdb session showing the problem.  It jumps around
a lot because it is at -O2.  What happens is that "getrlimit"
and "setrlimit" are each called twice -- once for RLIMIT_DATA
and once for RLIMIT_AS.  "atoi()" is then called to convert
a the string "1" to binary and that is used to select a "for"
loop.  The first iteration fails:

  for (repeat = 0; repeat < NUM_ROUNDS; repeat++)
    {
      /* This may produce a temporary memory allocation of 11000 bytes.
         but should not result in a permanent memory allocation.  */
      if (dprintf (STDOUT_FILENO, "%011000d\n", 17) == -1
          && errno == ENOMEM)
        return 1;
    }

So it seems the ltrace I sent before is correctly showing the problem
viz., the rpl_fprintf call is failing with ENOMEM.  The test appeared
to succeed because ltrace will exit 0 even if the traced program
exits 1.

The gdb session follows:

Breakpoint 1, main (argc=2, argv=0x7fffffffdc78)
    at ../../tests/test-dprintf-posix2.c:48
48      {
(gdb) n
82        if (getrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
48      {
(gdb) 
83          return 77;
(gdb) 
48      {
(gdb) 
82        if (getrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
84        if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max >
MAX_ALLOC_TOTAL)
(gdb) 
85          limit.rlim_max = MAX_ALLOC_TOTAL;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
86        limit.rlim_cur = limit.rlim_max;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
88          return 77;
(gdb) 
87        if (setrlimit (RLIMIT_DATA, &limit) < 0)
(gdb) 
94        if (getrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
96        if (limit.rlim_max == RLIM_INFINITY || limit.rlim_max >
MAX_ALLOC_TOTAL)
(gdb) 
97          limit.rlim_max = MAX_ALLOC_TOTAL;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
98        limit.rlim_cur = limit.rlim_max;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) p limit
$1 = {rlim_cur = 10000000, rlim_max = 10000000}
(gdb) n
100         return 77;
(gdb) 
99        if (setrlimit (RLIMIT_AS, &limit) < 0)
(gdb) 
103       arg = atoi (argv[1]);
(gdb) 
104       if (arg == 0)
(gdb) p arg
$2 = <value optimized out>
(gdb) p argv[1]
$3 = 0x7fffffffe210 "1"
(gdb) s
103       arg = atoi (argv[1]);
(gdb) n
104       if (arg == 0)
(gdb) s
120           if (dprintf (STDOUT_FILENO, "%011000d\n", 17) == -1
(gdb) n
121               && errno == ENOMEM)
(gdb) p errno
$4 = 12
(gdb) n
122             return 1;
(gdb) 
126     }
(gdb) c
Continuing.

Program exited with code 01.

-- 
Configure bugmail: http://sourceware.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.