low-mem can make glibc's stream output functions segfault

[Jim Meyering <jim at meyering dot net>]

FYI, I've just put the following in

  http://bugzilla.redhat.com/365111

------------------------
An aside,

Can any of you recommend a good malloc-replacement tool to automate
inducing such failures probabilistically?  I guess some might call it
a malloc fuzzer, but a few quick searches didn't turn up anything obvious.
------------------------

Here's the report:

glibc's stream output functions (e.g., printf, fputs, fwrite, etc.)
always allocate memory upon stream initialization.  The first output
appears to cause allocation of a 4KB block (a page).  I want to know if
that first output operation can fail without setting the stream error
indicator, like other ENOMEM failures do.  I tried to provoke this, to see
if a failure of that precise allocation would provoke an ferror-detectable
failure, but ran into something else.  When that particular mmap call
fails (I think it's the one in filedoalloc.c from the ALLOC_BUF macro),
it ends up causing a segfault 5 or 6 levels up the stack.

Glancing through the code, this seems to happen because
_IO_new_file_overflow isn't prepared for a NULL f->_IO_buf_base pointer.

  $ printf '#include <stdio.h>\nint main(){printf("foo");return 0;}\n' > k.c
  $ gcc -pthread -g -Wall -W -O k.c
  $ gdb -q ./a.out
  Using host libthread_db library "/lib64/libthread_db.so.1".
  (gdb) b printf
  Function "printf" not defined.
  Make breakpoint pending on future shared library load? (y or [n]) y

  Breakpoint 1 (printf) pending.
  (gdb) r
  Starting program: /t/a/a.out

  Breakpoint 2 at 0x3c5084ca00
  Pending breakpoint "printf" resolved

  Breakpoint 2, 0x0000003c5084ca00 in printf () from /lib64/libc.so.6
  (gdb) b mmap64
  Breakpoint 3 at 0x3c508d18d0
  (gdb) c
  Continuing.

  Breakpoint 3, 0x0000003c508d18d0 in mmap64 () from /lib64/libc.so.6
  (gdb) ret -1
  Make selected stack frame return now? (y or n) y

  #0  0x0000003c508613db in _IO_file_doallocate_internal () from /lib64/libc.so.6
  (gdb) p errno=12
  $2 = 12
  (gdb) c
  Continuing.

  Program received signal SIGSEGV, Segmentation fault.
  0x0000003c5086c8dc in _IO_new_file_overflow () from /lib64/libc.so.6
  (gdb) bt
  #0  0x0000003c5086c8dc in _IO_new_file_overflow () from /lib64/libc.so.6
  #1  0x0000003c5086ec34 in _IO_default_xsputn_internal () from /lib64/libc.so.6
  #2  0x0000003c5086d881 in _IO_new_file_xsputn () from /lib64/libc.so.6
  #3  0x0000003c50842f50 in vfprintf () from /lib64/libc.so.6
  #4  0x0000003c5084ca9a in printf () from /lib64/libc.so.6
  #5  0x00000000004004cb in main () at k.c:2
  (gdb)

I simulated the mmap64 failure by return -1 and setting errno=12.
12 is ENOMEM:

    $ e=ENOMEM; perl -le "use Errno '$e';print $e"
    12

This is on fedora rawhide.

  $ rpm -q glibc
  glibc-2.7-2