[ITP] util-linux

[Charles Wilson]

Eric Blake wrote:

> According to Yaakov S (Cygwin Ports) on 3/2/2006 3:06 PM:
>>>> No.  With Corinna's patch to add CYGWIN=transparent_exe option, we are
>>>> asking for problems if we distribute a script alongside an exe.
>> OK, but I can think of one *major* "violation" of this: libtool-built
>> applications, when linking against a to-be-installed library in the same
>> package.  In such a case, the real .exe is placed in .libs, and both an
>> extension-less libtool script  and a launcher .exe are placed in the
>> builddir; this allows one to run the application w/o installing it.
> 
> Yes, I'm aware of that, and I would love to spend some free time trying to
> solve another way for libtool to work without relying on the (subtle)
> difference between 'foo' and 'foo.exe' when testing uninstalled
> libtoolized applications.  This libtool (ab)use of filenames has already
> had difficulties with managed mounts and trailing dots.  Then there was
> the time when I tried to patch rm in coreutils-5.3.0 to handle .exe
> transparently (and immediately had to retract that change because of libtool).

For those following along at home, the issue behind libtool's "abuse" is 
as follows.  (Sidenote: for eons foo.exe != foo and libtool relied on 
that.  Later, changes in cygwin forced libtool to rely instead on 
foo.exe != foo. (with trailing dot).  More "improvements" to cygwin's 
exe handling [but at least they are optional, thank goodness] now break 
even THAT and it's LIBTOOL's fault for "abusing" longstanding behavior? 
I don't think so.)

On all platforms, libtool creates wrapper scripts for two purposes: 
first, to set environment variables (PATH, LD_PRELOAD, etc) before 
launching the "real" uninstalled executable, so that the uninstalled 
executable can find its (uninstalled) dependencies.  For instance,

mypackage/lib/ contains code for mylib shared library
mypackage/app/ contains code for myapp executable, which
                depends on mylib

After compile,
    the mylib.so shared library is in    mypackage/lib/.lib/  and
    the myapp    executable binary is in mypackage/src/.lib/
    the myapp    wrapper script is in mypackage/src/
and it will set LD_LIBRARY_PATH (or whatever variable is appropriate) to 
mypackage/lib/.lib, and then lauch mypackage/src/.lib/myapp.  All is well.

Except on cygwin, but we'll get to that later.

The OTHER reason for the wrapper script is it serves to hold persistent 
state for the target executable -- libtool itself *parses* the script to 
read some of those variable values in "the next" libtool invocation 
after compilation (e.g. libtool --mode=install).

So, what's the problem on cygwin?  Most makefiles -- especially those 
already tailored for cygwin or generated by autoconf'ing a Makefile.in, 
or automake'ing a Makefile.am->Makefile.in->Makefile, have their 
executable target names set as

myapp$(EXE_EXT): dependencies...
	rule....

When EXE_EXT= <empty>, as in most platforms, there is no problem.  The 
WRAPPER script is named "myapp" and the target rule is "myapp" so the 
wrapper satisfies the rule.  On cygwin, it does not: so make never 
believes that the rule has been satisfied, and always tries to relink 
the app. Over and over.

There are two solutions: (1) name the wrapper script myapp.exe instead 
of myapp.  (2) create a wrapper *binary* named myapp.exe and lives in 
myproject/app/ NOT myproject/app/.lib, and which either [a] sets all the 
important environment variables and then exec's the "real" executable, 
or [b] exec's the wrapper script, which sets the environment and 
launches the "real" executable.

(1) is simple, clean, beautiful, and wrong.  While current cygwin (at 
least, under the current bash shell; I didn't try others) will in fact 
treat the following:

----- foo.exe ----
#!/bin/sh
echo This is 'foo.exe' script
------------------

as an actual script, and use the sh-bang line to exec the correct 
interpreter, I do not think older cygwin's/shells did so.  I'm pretty 
sure msys/mingw will barf.  And I know it'll confuse users -- and if 
having the following two files in a directory:
    foo.exe
    foo
is considered "abuse" then it's absolutely true that having
    foo.exe
which is NOT a binary executable is "abuse".  (Also, I don't know how 
cygwin will treat a script named "foo.exe" on a remote share, 
with/without nosmbntsec, or local/remote with/without cygexec/exec mount 
options, ...  This is just ASKING for even more trouble than we've got now.)

(2)[a] is problematic, because 'libtool --mode=install' can't source it, 
plus it duplicates knowledge; the same settings are stored in two 
places: the wrapper script (or its renamed replacement 
'target-data-file') and in the binary wrapper.

More on this later.

(2)[b] So this is what we did.  And it worked.  Until other people broke 
it by changing cygwin.  So we fixed it.  And now we've got a new cygwin 
option (which I had NEVER heard about until now!) that breaks it again 
-- and the maintainer of coreutils is pushing for more widespread use.

E.g. pushing to deliberately break libtool

Eric, please don't *push* CYGWIN=transparent_exe until *somebody* 
implements the following libtool fix.  Sadly, it's too late for any such 
major change to get into libtool-2.0 because libtool is effectively in 
code-slush right now.  But, if *somebody* implements this fix I'll put 
it in cygwin's dist of libtool-2.0 and push for it to go into libtool-2.0.x.

The fix:
---------

Suppose that the following changes were made:

(1) the binary wrapper takes a command line argument that causes it to 
emit the current contents of the wrapper script on stdout

(2) the binary wrapper sets the environment variables itself, and then 
exec's the real binary executable directly instead of the wrapper 
script.  This should be relatively easy: just make a copy of **environ, 
modify the entry in that copy with startswith PATH= (adding both the 
.lib dir which contains the target exe, and the .lib dirs which contain 
the DLLs on which it depends), and then use
    execve(name-of-target-exe, ...)
instead of
    execv(name-of-wrapper, ...).
Watch out for memory allocation issues wrt to the old environ[x] entry 
for PATH and the new one, tho.

(3) libtool (on cygwin|mingw) would then no longer generate the wrapper 
script.  However, when it needs to source the contents of that script, 
it instead calls
    'wrapper.exe --emit-script > .lib/wrapper-env-settings
sources THAT, and then rm -f's it.  Note that (a) the env-settings 
script is both named something other than the actual name of the 
makefile target-without-.exe, AND lives in the .lib subdir.

-------

This would make everybody happy, wouldn't "duplicate" the storage of 
information, would keep the current good behavior of NOT relinking apps 
over-and-over, and could EVEN be extended across all platforms once it's 
working on cygwin/mingw.   But it's a lot of work; fortunately each of 
the three items can be implemented and tested separately.

But, until these changes happen (and I'm not likely to have time to do 
them myself, although PTC and tested), my answer to complaints is going 
to be a pointer to this post, and "don't use transparent_exe".

--
Chuck