Re: Building glibc 2.19 for OS/ABI UNIX - System V

["Carlos O'Donell" <carlos at systemhalted dot org>]

On Thu, Jul 24, 2014 at 3:44 PM, Shaun Jackman <sjackman@gmail.com> wrote:
> I'd like to run a binary executable that was compiled on a newer
> system (Ubuntu 14.04) on an older system (CentOS 5.10). To that end,
> I'm compiling glibc 2.19 on the older system (CentOS 5.10). I realize
> that the short answer is probably "Don't do that". All the same, call
> it a learning experience for me. If it doesn't work and can't work,
> I'd like to learn first hand why it doesn't work.

It is not my place to judge your requirements or needs, your question
is a perfectly valid one, and I have some answers for you.

To run a newer application on an older system requires *all* of
runtimes that are needed by the newer application.

It does work, and I've done things like run Chrome 30 on RHEL6 but it
requires 35+ newer libraries, changing the scripts to call the new
ld.so, and because I don't want to modify the chrome binaries
(licensing reasons) I preload a DSO that interposes all of the exec*
family of functions to run any helper programs under the right
runtimes.

> The current problem I'm facing is the error message `ELF file OS ABI
> invalid` because the `libc-2.19.so` that I've compiled on CentOS 5.10
> uses an ABI unsupported by the system's loader `/lib64/ld-2.5.so`. Is
> it possible to get past this without using a newer loader?

Yes it is possible to revert the OS ABI changes, but that doesn't
solve your problem.

Let me reiterate why.

All of the core runtime, loader, libc, libm, libpthread, libdl, etc,
which are built by glibc, must be upgraded *together* or downgraded
*together*. They are coordinating between each other to provide the
implementation and it is unsupported to mix and match.

> Using a newer loader, although possible, would make running the new
> executable(s) pretty cumbersome. I could either wrap all the new
> executables in a shell script that runs the new loader, pretty ugly,
> or I could patch the ELF header to specify the new dynamic linker (the
> effect of `ld --dynamic-linker`). I'd prefer to use the old loader, if
> possible.

In order of least isolation to most isolation you have the following options:

(a) Copy new versions of all the new libraries you need to your
system. Rewriting the program header e.g. INTERP and adding DT_RPATH
to force the application to search in a special place first. This
option is great since you use the dyanmic loader to your advantage to
ensure the right libraries are used.

(b) Copy new versions of all the new libraries you need to your
system. Wrap all executables in shell scripts that invoke them via the
new loader and the right `--library-path` options, possibly preloading
a shared library that interposes exec*. To give you an idea of what an
interposer library would look like see the attached exec_wrapper.c (it
may make your eyes bleed) that I've used to run chrome in a contained
runtime. This option is harder but allows you to avoid changing any
binaries if licensing is a problem.

(c) Setup a Ubuntu chroot, chroot into it, and run your application.

(d) Setup a linux container (cgroup) with a Ubuntu chroot, and run
your application in there.

(e) Setup a Ubuntu VM and run your application there.

Lastly, the core runtimes are backwards compatible, and we work very
very hard to keep it that way (at least on x86*). You could upgrade
glibc to a newer glibc from say Fedora, but at that point you might
want to stop compiling applications on that system. Any newly compiled
applications might possibly use newer symbols from the newer glibc
(contaminated with a newer ABI) and you could never run those
applications on other Cent OS 5.10 boxes because they would not have
the newer glibc installed. However, nothing stops you from installing
a new core runtime... except that we don't guarantee bug backwards
compatibility or undefined behaviour compatibility, so some other old
applications may stop working (despite the fact that glibc is
correct).

The question of running newer applications on older distributions is a
very old question, and one that has subtle tradeoffs.

The industry believes the ultimate solution is going to be containers,
but that's just a synonym for "static linking" and has it's own
problems e.g. all your containers, not just all of your systems, need
updating if there is a security bug in a shipped component. So it's
much more like a light-weight VM.

In full disclosure, I work for Red Hat and I am the team lead for
glibc within the base OS tools team. I could talk all day about these
problems if you're interested, but I figure you've got a specific
problem to solve :-)

Cheers,
Carlos.

/* exec* interposer to run chrome.
   Copyright (C) 2014 Free Software Foundation, Inc.
   This file is part of the GNU C Library.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
   Lesser General Public License for more details.

   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, see
   <http://www.gnu.org/licenses/>.  */

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <dlfcn.h>
#include <string.h>
#include <unistd.h>
#include <limits.h>

/* Enable debugging for now. Set to 0 to disable debugging.  */
#define DEBUG 0

#define BUFSZ 1024

#define FN(ptr, type, name, args) ptr = (type (*)args)dlsym (RTLD_NEXT, name)

char *progs[] = { 
#define P_CHROME 0
	"/opt/google/chrome/chrome", 
#define P_SANDBOX 1
	"/opt/google/chrome/chrome-sandbox",
#define P_NACLH 2
	"/opt/google/chrome/nacl_helper" };
int nprogs = 3;

char ldso[] = "/opt/google/chrome/ld-linux-x86-64.so.2";
char ldso_a1 [] = "--library-path";
char ldso_a2 [] = "/opt/google/chrome";

/* Prep to run ld.so.  */
#define rewrite_with_ldso() \
({					\
	strcpy (newfilename, ldso);	\
	newargv[0] = ldso;		\
	newargv[1] = ldso_a1;		\
	newargv[2] = ldso_a2;		\
})

#define copy_args_to_output() \
({					\
	while (argv[k] != NULL)		\
	  {				\
	    newargv[j++] = argv[k++];	\
	  }				\
	newargv[j] = NULL;		\
})

/* Copy the filename and argv to execute into the new* variants.  */
#define copy_input_to_output() \
({									\
  strcpy (newfilename, filename);					\
  while (argv != NULL && argv[i] != NULL)				\
  {									\
    snprintf (buf, BUFSZ, PREFIX " argv[%d] = %s\n", i, argv[i]);	\
    write (fd, buf, strlen (buf));					\
    newargv[j++] = argv[i++];						\
  }									\
  newargv[j] = NULL;							\
})

/* Handle running the supported programs under the new runtimes.  */
#define rewrite_with_new_runtime() \
({										\
  /* This is chrome using /proc/self/exe. Fixup name.  */			\
  if ((strcmp (filename, "/proc/self/exe") == 0)				\
      && (strncmp (argv[1], "--type", strlen ("--type")) == 0))			\
    {										\
      snprintf (buf, BUFSZ, PREFIX " chrome called %s by accident.\n", ldso);	\
      write (fd, buf, strlen (buf));						\
      filename = progs[P_CHROME];						\
    }										\
										\
  for (i = 0; i < nprogs; i++)							\
    if (strcmp (progs[i], filename) == 0)					\
      {										\
	/* Running one of the special programs! Use special runtime.  */	\
	rewrite_with_ldso();							\
	newargv[3] = progs[i];							\
	j = 4;									\
	k = 1;									\
	copy_args_to_output();							\
	snprintf (buf, BUFSZ, PREFIX " Override %s\n", filename);		\
	write (fd, buf, strlen (buf));						\
      }										\
										\
  /* This is chrome using the wrong argv[0]. Fixup arguments.  */		\
  if ((strcmp (filename, progs[P_SANDBOX]) == 0)				\
      && (strcmp (argv[1], ldso) == 0))						\
    {										\
      snprintf (buf, BUFSZ, PREFIX " chrome called %s by accident.\n", ldso);	\
      write (fd, buf, strlen (buf));						\
      newargv[4] = progs[P_CHROME];						\
    }										\
										\
  snprintf (buf, BUFSZ, PREFIX " %s\n", newfilename);				\
  write (fd, buf, strlen (buf));						\
})

#if DEBUG
# define OUTPUTFILE "/opt/google/chrome/libexec_wrapper.out"
#else
# define OUTPUTFILE "/dev/null"
#endif

int
execve(const char *filename, char *const argv[], char *const envp[])
{
  int i = 0, j = 0, k, fd;
  char buf[BUFSZ];
  char *newargv[255];
  char newfilename[255];

  static int (*func)(const char *, char **, char **);
  FN(func,int,"execve",(const char *, char **const, char **const));
#undef PREFIX
#define PREFIX "[execve]"

  fd = open (OUTPUTFILE, O_WRONLY | O_CREAT | O_APPEND);
#if DEBUG
  chmod (OUTPUTFILE, S_IRWXU | S_IRWXG | S_IRWXO);
#endif

  copy_input_to_output();
  rewrite_with_new_runtime();

  return (*func) (newfilename, (char**) newargv, (char **) envp);
}

int
execv(const char *filename, char *const argv[])
{
  int i = 0, j = 0, k = 0, fd;
  char buf[BUFSZ];
  char *newargv[255];
  char newfilename[255];

  static int (*func)(const char *, char **);
  FN(func,int,"execv",(const char *, char **const));
#undef PREFIX
#define PREFIX "[execv]"

  fd = open (OUTPUTFILE, O_WRONLY | O_CREAT | O_APPEND);
#if DEBUG
  chmod (OUTPUTFILE, S_IRWXU | S_IRWXG | S_IRWXO);
#endif

  copy_input_to_output();
  rewrite_with_new_runtime();

  close (fd);

  return (*func) (newfilename, (char **) newargv);
}

int
execvp (const char *filename, char *const argv[])
{
  int i = 0, j = 0, k = 0, fd;
  char buf[BUFSZ];
  char *newargv[255];
  char newfilename[255];

  static int (*func)(const char *, char **);
  FN(func,int,"execvp",(const char *, char **const));
#undef PREFIX
#define PREFIX "[execvp]"

  fd = open (OUTPUTFILE, O_WRONLY | O_CREAT | O_APPEND);
#if DEBUG
  chmod (OUTPUTFILE, S_IRWXU | S_IRWXG | S_IRWXO);
#endif

  copy_input_to_output();
  rewrite_with_new_runtime();

  close (fd);

  return (*func) (newfilename, (char **) newargv);
}