Add XDR support (only built #if cygwin, for now)

[Charles Wilson <cygwin at cwilson dot fastmail dot fm>]

The attached patch(es) add XDR support to newlib. eXternal Data
Representation (XDR) is a standard data encoding mechanism (RFC 1832,
4506) most frequently seen in the implementation of the RPC protocol.
XDR allows binary data structures to be encoded in an architecture
independent manner for communication between heterogeneous computer systems.

The XDR functions are provided by glibc, and I think it would be helpful
to (optionall) provide them within newlib, as well. As DK stated on the
cygwin list: "'Small embedded system' doesn't mean 'not
network-connected' any more"

For years, the Sun RPC code, and the XDR implementation, was in legal
license limbo
  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=181493
as its license terms, while open, were of debatable compatibility with
the GPL. In February of 2009, that changed:

http://blogs.sun.com/webmink/entry/old_code_and_old_licenses
http://lwn.net/Articles/319648/

As documented in the libtirpc rpm.spec file from Fedora 11:
* Tue May 19 2009 Tom "spot" Callaway <xxxx@redhat.com> 0.1.10-7
- Replace the Sun RPC license with the BSD license, with the explicit
  permission of Sun Microsystems

So, the attached implementations were taken from Fedora 11's libtirpc
package, after the modifications by Tom Callaway. Thus, each file
carries the (3-clause) BSD license, and not the old "SunRPC" license.

Last year, I worked on it some to get it to compile cleanly under cygwin
(1.5,1.7), mingw, msvc80, and linux (32bit, 64bit), while being careful
to maintain the BSD license.  I also wrote a whole slew of tests to
guarantee correct operation and identical representation on all of these
platforms (these tests carry the MIT/X license). I won't say that I
created full test coverage, but it's awfully close.  The result of that
effort:

http://code.google.com/p/bsd-xdr/

The attached patch is basically a fork of that bsd-xdr core code. I
removed all the #ifdefs that made msvc80 and mingw happy, as they aren't
needed in newlib and would complicate any attempt to expand the use of
the new code to newlib targets other than cygwin.  I reworked both the
headers and implementation files to follow newlib-isms like using
_EXFUN() and _DEFUN macros and such.  I did not, however, attempt to add
'chew'-able documentation to the various files. If desired, I could work
on that as a series of follow-on patches.

This fork, unlike the SunRPC/bsd-xdr implementation, does not rely on
<byteorder.h> (which isn't available inside newlib) for ntohl/htonl.
Instead, it uses its own private ntohl/htonl implementations.  Also,
while most xdr implementations print errors to stderr using the
BSD-flavor warnx() function, this version allows the caller to configure
an error reporting function (more below).

I did not include the test suite; it would take quite a bit of effort to
translate the bsd-xdr/test/ code into cygwin's testsuite framework --
and newlib/libc/ itself doesn't appear to have a test suite AFAICT.  I'm
not sure it's worth the effort, actually -- none of the BSD's or glibc
or linux's include test code for XDR, which is why I had to code tests
from the ground up for my bsd-xdr package.  I did link my test suite
against this implementation inside a newly-built cygwin1.dll that
exports these new XDR functions, and it passed with flying colors.

Obviously, the newlib contributions retain their original -- that is,
BSD -- licenses, like all the other such files added to newlib.
(xdr_private.h/.c are MIT/X).

The XDR code is currently built only if $host==cygwin. On all other
platforms the only effect of this patch is that two new headers,
rpc/types.h and rpc/xdr.h, are installed. (newlib installs stdio.h even
if stdio is "turned off", so I thought to follow that pattern).


Error reporting:
================
As mentioned above, most XDR implementations use the BSDish warnx
function to immediately print the (very rare) error messages to stderr.
Similarly, here error messages are printed to stderr by default --
except that a framework is provided for the caller to register a
different callback function for error reporting. (The cygwin glue, not
included here, registers a callback that redirects errors to cygwin's
debug strace facility).

This registration function is not declared in the public headers; I
expect that, if any platform other than cygwin "turns on" xdr, will also
declare and auto-register an appropriate callback, here inside newlib,
so it can simply include xdr_private.h to get the necessary decls.

I didn't worry at all about multi-thread issues with regards to this
error reporting callback function.

I'm of mixed feelings about this whole error reporting issue: all of the
error messages, inherited from the original SunRPC code, are "out of
memory" errors. Perhaps it would be better to avoid reporting these
messages to any console/stream, and instead silently return failure with
errno=ENOMEM?  OTOH, the SunRPC implementation is almost universal...


Applicability to newlib patforms other than cygwin
==================================================
At present, XDR is not built unless $host==cygwin.  I figure other
interested parties can later activate it when they put in whatever
minimal effort may be required to get it to compile for them. (It's
pretty dirt-simple C code; build system integration -- already done --
and header file prereqs are actually the hard part).  Other targets
might need to #ifdef out support for e.g. long longs or doubles; maybe
add some intelligence to optionally exclude those limited portions of
the XDR code that require stdio to work...but as-is this XDR
implementation is 90% of the way there for those platforms (and 100% of
the way there for cygwin).  As submitted though, XDR is completely
"turned off" for all but $host cygwin.

I recognize that embedded targets can sometimes be picky about
implementation details (is malloc() allowed? are stdio streams
available?, etc).  As provided here, XDR uses both.  However, xdr (and
rpc) code is basically 25 years old or more, and little changed in
implementation over that time.  If embedded targets want an
implementation that -- for instance -- avoids even the minimal use of
malloc() required by this "standard" implementation, they can put one in
libc/sys/$arch/, and this implementation can be a good starting point
(*).  But for most systems with less draconian rules, the standard
implementation should be fine, with the addition of some possible
#ifdefs around longlong, double, and/or stdio-requiring bits.

(*) Besides, well-designed protocols based on XDR encode/decode should
specify the maximum length of "variable length" data, which in turn
allows pre-allocated storage to be used when ACTUALLY
serializing/deserializing data.  In this way, even embedded systems with
"draconian" malloc rules can guarantee that the calls to malloc()/free()
are never actually encountered in practice.


The patch(es)
=============
I've split the patch into two pieces: the first are the modifications to
existing files plus all the brand new files, while the second is the
result of regenerating the auto* files (Makefile.in, etc).  Both patches
are extremely large, so they are attached compressed.  I've pasted
inline the portion of the first patch that /modifies/ existing files, so
you can see how the XDR support is integrated into the newlib framework.

Part 1: xdr-newlib-primary.patch
2010-02-24  Charles Wilson  <...>

        Add eXtensible Data Record (XDR) support
	* configure.host: Build libc/xdr only on cygwin.
	* Makefile.am: Install xdr headers.
	* libc/configure.in: Support new libc/xdr subdirectory.
	* libc/Makefile.am: Support new libc/xdr subdirectory.
	* libc/include/rpc/types.h: New.
	* libc/include/rpc/xdr.h: New.
	* libc/xdr/Makefile.am: New.
	* libc/xdr/dummy.c: New.
	* libc/xdr/xdr.c: New.
	* libc/xdr/xdr_array.c: New.
	* libc/xdr/xdr_float.c: New.
	* libc/xdr/xdr_mem.c: New.
	* libc/xdr/xdr_private.c: New.
	* libc/xdr/xdr_private.h: New.
	* libc/xdr/xdr_rec.c: New.
	* libc/xdr/xdr_reference.c: New.
	* libc/xdr/xdr_sizeof.c: New.
	* libc/xdr/xdr_stdio.c: New.

Part 2: xdr-newlib-auto.patch
2010-02-24  Charles Wilson  <...>

	Regenerate using ac-2.63 and am-1.11.1
	* libc/xdr/Makefile.in: New.
	* Makefile.in: Regenerate.
	* libc/configure: Regenerate.
	* libc/Makefile.in: Regenerate.
	* libc/argz/Makefile.in: Regenerate.
	* libc/ctype/Makefile.in: Regenerate.
	* libc/errno/Makefile.in: Regenerate.
	* libc/iconv/ccs/binary/Makefile.in: Regenerate.
	* libc/iconv/ccs/Makefile.in: Regenerate.
	* libc/iconv/ces/Makefile.in: Regenerate.
	* libc/iconv/lib/Makefile.in: Regenerate.
	* libc/iconv/Makefile.in: Regenerate.
	* libc/locale/Makefile.in: Regenerate.
	* libc/misc/Makefile.in: Regenerate.
	* libc/posix/Makefile.in: Regenerate.
	* libc/reent/Makefile.in: Regenerate.
	* libc/search/Makefile.in: Regenerate.
	* libc/signal/Makefile.in: Regenerate.
	* libc/stdio/Makefile.in: Regenerate.
	* libc/stdio64/Makefile.in: Regenerate.
	* libc/stdlib/Makefile.in: Regenerate.
	* libc/string/Makefile.in: Regenerate.
	* libc/syscalls/Makefile.in: Regenerate.
	* libc/time/Makefile.in: Regenerate.
	* libc/unix/Makefile.in: Regenerate.

The portion of xdr-newlib-primary.patch that modifies existing files,
pasted inline. Probably kills word wrapping, but it's all included in
xdr-newlib-primary.patch.lzma anyway.



diff -u src-newlib-p/newlib/configure.host src/newlib/configure.host
--- src-newlib-p/newlib/configure.host	2010-01-01 20:21:24.262000000 -0500
+++ src/newlib/configure.host	2010-02-12 17:46:46.261400000 -0500
@@ -55,6 +55,7 @@
 signal_dir=signal
 stdio_dir=stdio
 stdio64_dir=
+xdr_dir=
 syscall_dir=
 unix_dir=
 mach_add_setjmp=
@@ -349,6 +350,7 @@
   *-*-cygwin*)
 	posix_dir=posix
 	stdio64_dir=stdio64
+	xdr_dir=xdr
 	;;
   *-*-netware*)
 	signal_dir=
diff -u src-newlib-p/newlib/Makefile.am src/newlib/Makefile.am
--- src-newlib-p/newlib/Makefile.am	2009-09-14 01:08:23.490000000 -0400
+++ src/newlib/Makefile.am	2010-02-12 20:28:52.601400000 -0500
@@ -269,6 +269,10 @@
 	     $(INSTALL_DATA) $$i
$(DESTDIR)$(tooldir)/include/machine/`basename $$i`; \
 	    else true; fi ; \
 	  done; \
+	  $(mkinstalldirs) $(DESTDIR)$(tooldir)/include/rpc; \
+	  for i in $(srcdir)/libc/include/rpc/*.h; do \
+	   $(INSTALL_DATA) $$i $(DESTDIR)$(tooldir)/include/rpc/`basename $$i`; \
+	  done; \
 	  $(mkinstalldirs) $(DESTDIR)$(tooldir)/include/sys; \
 	  for i in $(srcdir)/libc/include/sys/*.h; do \
 	   $(INSTALL_DATA) $$i $(DESTDIR)$(tooldir)/include/sys/`basename $$i`; \
diff -u src-newlib-p/newlib/libc/configure.in src/newlib/libc/configure.in
--- src-newlib-p/newlib/libc/configure.in	2010-02-12 16:59:18.157600000
-0500
+++ src/newlib/libc/configure.in	2010-02-12 17:43:07.823400000 -0500
@@ -94,6 +94,20 @@
 AC_SUBST(LIBC_STDIO64_DEF)
 AM_CONDITIONAL(HAVE_STDIO64_DIR, test x${stdio64_dir} != x)

+LIBC_XDR_LIB=
+LIBC_XDR_DEF=
+if test -n "${xdr_dir}"; then
+  if test "${use_libtool}" = "yes"; then
+    LIBC_XDR_LIB=${xdr_dir}/lib${xdr_dir}.${aext}
+  else
+    LIBC_XDR_LIB=${xdr_dir}/lib.${aext}
+  fi
+  LIBC_XDR_DEF=${xdr_dir}/stmp-def
+fi
+AC_SUBST(LIBC_XDR_LIB)
+AC_SUBST(LIBC_XDR_DEF)
+AM_CONDITIONAL(HAVE_XDR_DIR, test x${xdr_dir} != x)
+
 LIBC_SYSCALL_LIB=
 if test -n "${syscall_dir}"; then
   if test "${use_libtool}" = "yes"; then
@@ -171,5 +185,5 @@
 AC_SUBST(LIBC_MACHINE_LIB)
 AC_SUBST(machine_dir)

-AC_CONFIG_FILES([Makefile argz/Makefile ctype/Makefile errno/Makefile
locale/Makefile misc/Makefile reent/Makefile search/Makefile
stdio/Makefile stdio64/Makefile stdlib/Makefile string/Makefile
time/Makefile posix/Makefile signal/Makefile syscalls/Makefile
unix/Makefile iconv/Makefile iconv/ces/Makefile iconv/ccs/Makefile
iconv/ccs/binary/Makefile iconv/lib/Makefile])
+AC_CONFIG_FILES([Makefile argz/Makefile ctype/Makefile errno/Makefile
locale/Makefile misc/Makefile reent/Makefile search/Makefile
stdio/Makefile stdio64/Makefile stdlib/Makefile string/Makefile
time/Makefile posix/Makefile signal/Makefile syscalls/Makefile
unix/Makefile iconv/Makefile iconv/ces/Makefile iconv/ccs/Makefile
iconv/ccs/binary/Makefile iconv/lib/Makefile xdr/Makefile])
 AC_OUTPUT
diff -u src-newlib-p/newlib/libc/Makefile.am src/newlib/libc/Makefile.am
--- src-newlib-p/newlib/libc/Makefile.am	2010-02-12 15:21:53.238200000 -0500
+++ src/newlib/libc/Makefile.am	2010-02-12 17:53:34.425000000 -0500
@@ -34,10 +34,15 @@
 NEWLIB_ICONV_LIBS =
 endif

+if HAVE_XDR_DIR
+XDR_SUBDIR = xdr
+endif
+
 # The order of SUBDIRS is important for the integrated documentation.
 # Do not change the order without considering the doc impact.
 SUBDIRS = argz stdlib ctype search $(STDIO_SUBDIR) $(STDIO64_SUBDIR)
string $(SIGNAL_SUBDIR) time locale sys reent \
-	$(extra_dir) errno misc machine $(UNIX_SUBDIR) $(POSIX_SUBDIR)
$(SYSCALLS_SUBDIR) $(NEWLIB_ICONV_DIRS) .
+	$(extra_dir) errno misc machine $(UNIX_SUBDIR) $(POSIX_SUBDIR)
$(SYSCALLS_SUBDIR) $(NEWLIB_ICONV_DIRS) \
+	$(XDR_SUBDIR) .

 noinst_DATA = $(CRT0)

@@ -50,6 +55,7 @@
 	search/libsearch.$(aext) \
 	$(LIBC_STDIO_LIB) \
 	$(LIBC_STDIO64_LIB) \
+	$(LIBC_XDR_LIB) \
 	string/libstring.$(aext) \
 	$(LIBC_SIGNAL_LIB) \
 	time/libtime.$(aext) \
@@ -72,6 +78,7 @@
 	search/lib.$(aext) \
 	$(LIBC_STDIO_LIB) \
 	$(LIBC_STDIO64_LIB) \
+	$(LIBC_XDR_LIB) \
 	string/lib.$(aext) \
 	$(LIBC_SIGNAL_LIB) \
 	time/lib.$(aext) \
@@ -125,6 +132,7 @@
 	ctype/stmp-def \
 	$(LIBC_STDIO_DEF) \
 	$(LIBC_STDIO64_DEF) \
+	$(LIBC_XDR_DEF) \
 	string/stmp-def \
 	$(LIBC_SIGNAL_DEF) \
 	time/stmp-def \

Attachment: xdr-newlib-primary.patch.lzma
Description: Binary data

Attachment: xdr-newlib-auto.patch.lzma
Description: Binary data