dynamic loading of so's on clusters

[Peter Bowen <pzb at datastacks dot com>]

On beowulf clusters, it in not uncommon to have a master node with a full
set of libraries, while the compute nodes have few, if any.  One way to
accomplish this is by using BProc, the beowulf distributed process
space.  BProc is written by Erik Hendriks at Los Alamos National Lab,
adn is used by Scyld, Paralogics, and Linux Networx in their cluster
products.  It adds a small amount of code to glibc to provide a fallback
mechanism  if the library can't be found on a compute node.  The fallback
is to ask BProc to get the library, if possible, from the master node.
Right now the patch is small, and specific to BProc, but other systems,
such as mosix, may want similar functionality in the future.

It would be nice to see something integrated into glibc to
support single system image clusters.  I am attaching the bproc patch as a
starting point.  If there is a better way to implement this functionality,
I would be interested.

Thanks.
Peter


$Id: glibc-2.2.3-bproc-fr.patch,v 1.2 2001/08/21 21:59:54 hendriks Exp $

--- glibc-2.2.3/elf/dl-load.c.orig	Tue Aug 21 15:56:00 2001
+++ glibc-2.2.3/elf/dl-load.c	Tue Aug 21 15:57:03 2001
@@ -1188,6 +1188,25 @@
     _dl_debug_printf_c ("\t\t(%s)\n", what);
 }
 
+
+/* THIS IS COMPLETELY DISGUSTING.. but here's a bit of the bproc library...
+ *
+ * blame: Erik Hendriks <erik@hendriks.cx>
+ */
+#if defined(__i386__)
+#define __NR_bproc 223
+#elif defined(__sparc__)
+#define __NR_bproc 211
+#elif defined(__alpha__)
+#define __NR_bproc 291
+#endif
+#define BPROC_SYS_FILEREQ 0x305
+
+static int 
+bproc_request_file(const char *name) {
+  return syscall(__NR_bproc, BPROC_SYS_FILEREQ, name);
+}
+
 /* Open a file and verify it is an ELF file for this architecture.  We
    ignore only ELF files for other architectures.  Non-ELF files and
    ELF files with different header information cause fatal errors since
@@ -1224,6 +1243,13 @@
 
   /* Open the file.  We always open files read-only.  */
   fd = __open (name, O_RDONLY);
+  if (fd == -1) {
+    int tmp;
+    tmp = errno;
+    fd = bproc_request_file(name);
+    if (fd == -1)
+      errno = tmp;
+  }
   if (fd != -1)
     {
       ElfW(Ehdr) *ehdr;
@@ -1415,21 +1441,6 @@
 	    {
 	      if (fd != -1)
 		this_dir->status[cnt] = existing;
-	      else
-		{
-		  /* We failed to open machine dependent library.  Let's
-		     test whether there is any directory at all.  */
-		  struct stat64 st;
-
-		  buf[buflen - namelen - 1] = '\0';
-
-		  if (__xstat64 (_STAT_VER, buf, &st) != 0
-		      || ! S_ISDIR (st.st_mode))
-		    /* The directory does not exist or it is no directory.  */
-		    this_dir->status[cnt] = nonexisting;
-		  else
-		    this_dir->status[cnt] = existing;
-		}
 	    }
 
 	  /* Remember whether we found any existing directory.  */