This is the mail archive of the cygwin@cygwin.com mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Glib2


[Back on list, just to make sure it's archived. Max and I have bounced some emails back-n-forth privately, so you're coming in on the middle of a conversation... See Gen Zhang's original message on 12/4/2002]

Also, this is kinda stream of conciousness...because I end up at the end of the message somewhere other than where I thought I was going... <g>

Okay, from Zhang's report:
> After re-libtoolize-ing the sources (to allow it to produce shared
> libs) the compilation is fine through glib and will produce
> libglib-2.0.la, but will complain about undefined references to
> things like '_g_free' and '_g_log' (which I presume is in
> libglib-2.0.la). I've attached the output from make (upon a second
> run, after it going through glib).

The error was not in creating the DLL in the first place(*). The error was

> /bin/bash ../libtool --mode=link gcc -g -O2 -Wall -D_REENTRANT -o
> glib-genmarshal.exe glib-genmarshal.o ../glib/libglib-2.0.la -liconv
> -lintl -liconv
> gcc -g -O2 -Wall -D_REENTRANT -o .libs/glib-genmarshal.exe
> glib-genmarshal.o ../glib/.libs/libglib-2.0.dll.a -L/usr/lib
> -L/lib/w32api -luser32 -lkernel32 /usr/lib/libintl.dll.a
> /usr/lib/libiconv.dll.a -Wl,--rpath -Wl,/usr/local/lib
> glib-genmarshal.o(.text+0x1807): In function `complete_in_arg':
> /usr/src/glib-2.0.7/gobject/glib-genmarshal.c:218: undefined reference
> to `_g_log'
> ...

(*) unless the DLL itself exported no symbols at all. But, new libtool DOES NOT USE def files anymore. AT ALL. (and, you can't even give it one as an argument...)

Max Bowsher wrote:

>>> .def files are passed as -Wl,-retain-symbols-file deffile, not,
>>> as is correct, the first 'object' file.

Which means this ^^^^^ is wrong.

"-Wl,--retain-symbols-file -Wl,filename" is perfectly okay -- see the ld manpage. However, $export_symbols contains a filename whose contents are a list of exports -- like a def file, but not exactly; it's more unix-focused. If you trace the code in ld.exe, you find that --retain-symbol-file's argument is handled in a COMPLETELY different manner than a def file would be (it mainly acts to prevent the given symbols from being stripped; it doesn't "trigger" the necessary hooks to export those symbols in a DLL on cygwin/mingw). So, --retain-symmbols-file is okay on other unices, but not really the right switch for libtool to pass to ld, on cygwin.

On cygwin, what you end up with is
"don't strip these symbols"
and
"there is no def file, so by default, export all symbols(*)"

(*) unless there are symbols that have been specifically decorated with __declspec(dllexport), in which case ONLY those decorated symbols are exported. It turns out that THIS is the root of the problem with glib+cygwin.

So, on cygwin, this definition of archive_expsym_cmds is probably wrong -- but not in the way you thought. Because **archive_expsym_cmds** is NOT USED (much). [except in glib...sigh...]

Cygwin defines "always_export_symbols" as "no" (unless you use the '--export-symbols <FILENAME>' switch when calling libtool), and "export_symbols_regex" is empty unless you explicitly call libtool with the '--export-symbols-regex <REGEX>' switch. Given those typical definitions, we use "archive_cmds" NOT "archive_expsym_cmds" almost 100% of the time. Only when either '--export-symbols <FILENAME>' or '--export-symbols-regex <REGEX>' are specified do we, on cygwin, use "archive_expsym_cmds".

And glib does. It uses '--export-symbols glib.def' (on OS_WIN32, but not CYGWIN). More below.

Further, as far as ld is concerned, the def file doesn't need to be the FIRST object file. That was the case for dlltool (or you could explicitly say "--def def.file"), but not for ld once the functionality was absorbed. For ld, the def file just needs to appear as one of the arguments to ld. (see gld_${EMULATION_NAME}_unrecognized_file in binutis/ld/emultempl/pe.em)

> But I'm worried about just assuming that the -export-symbols file is
> a def file. There was some code in libtool somewhere (recently
> removed) which searched for EXPORTS to try and determine if it was a
> def file, and else tries to make one.
>
> I'm not sure how export-symbols works on other platforms.

Explained above. Now, IF someone were to specify --export-symbols-regex or --export-symbols on cygwin, then the archive_expsyms_cmd would be used -- and we need something like

_LT_AC_TAGVAR(archive_expsym_cmds, $1)='$CC -shared -nostdlib
$predep_objects $libobjs $deplibs $postdep_objects $compiler_flags -o
$output_objdir/$soname $export_symbols
${wl}--out-implib,$lib'

(e.g. no retain-symbols argument, just include $export_symbols as one of the "object" files)

There's just one problem: ld tests "unknown objects" filenames to see if they end in ".def" or ".DEF". $export_symbols is (usually) libname.exp. Badness.

So, for this to work as we expect, we must
(a) change $export_symbols to "libname.def" not "libname.exp". Old versions of libtool (back in dlltool days) did
cp $export_symbols $output_objdir/$soname-def
(dlltool was less picky about the actual name of the .def file; you'd just say "--def foo" and it was happy. We'd need to
cp $export_symbols $output_objdir/$soname.def
to insure that the file ended with .def.

(b) change $export_symbols_cmds so that it prepends "EXPORTS". Now, since always_export_symbols is false by default, the only time $export_symbols_cmds ever gets called is when you need to GENERATE an exports list -- and since '--export-symbols <FILENAME>' takes an argument, you don't need to generate it in that case (it's in <FILENAME>). So, the only time, on cygwin, that $export_symbols_cmds is ever called, is when you've done '--export-symbols-regex <REGEX>' Which makes things EVER so much more complicated. Sigh.

Anyway, perhaps something like $export_symbols_cmds =
"\$NM \$libobjs \$convenience | \$global_symbol_pipe | \$SED 's/.* //'
| sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"
(note the final $SED command)

However, that still doesn't differentiate between DATA and function exports; the .def file, if used, must identify DATA exports. Which means $export_symbols_cmds has be a LOT more complicated:

$NM $libobjs $convenience | \$global_symbol_pipe | \
\$SED -e '/^[BCDG] /s/^\([BCDG]\) \(_[_A-Za-z0-9]*\) \([_A-Za-z0-9]*\)/\3 DATA/' | \
\$SED -e '/^[AISTW] /s/^\([AISTW]\) \(_[_A-Za-z0-9]*\) \([_A-Za-z0-9]*\)/\3/' | \
sort | uniq | \$SED -e '1iEXPORTS' > \$export_symbols"

Old libtools used to use a specialized program (impgen) to do this, and a combination of dlltool calls. But we want to stay away from that, and stick with ld, gcc, and nm. So, we'll have to work harder...

Worse, if you actually specify --export-symbols=FOO on the libtool command line (e.g "use the file 'FOO', which already exists and specifies the symbols to export, as $export_symbols") then $export_symbols_cmds is NOT called, AND you lose platform cross-compatibility in the $export_symbols file. On unix, the file $export_symbols should just contain
bob
fred
But on windows, it should contain
EXPORTS
bob
fred DATA
because we're using it (in my "proposal" above) as a def file.

Basically, what you're trying to do is build an .def file on the fly. This USED to be very helpful -- before the auto-import, auto-export functionality was added to ld.exe itself. Now, it just messes things up on cygwin. (It might still be useful on other platforms, but not here).

I think this whole concept (on-the-fly def file created by libtool) was a kludge, because dlltool NEEDED a def file (during its 'make a dll' mode), so libtool created one (by calling dlltool in its 'make a def file' mode). Now, we don't use dlltool anymore, and ld can automatically figure out what, and how (DATA or function) to export. So none of this is needed on windows anymore, in general.

The only time you need any of this "specially restrict the export list or specifically export a known list of symbols" is when you're doing something *special*. And that requires developer intervention....

In cygwin/mingw, the typical library needs to do one of the following:
1) export everything (almost always, this is what you want)
2) export everything EXCEPT ...
3) export only ...
4) export everything, BUT you've specifically decorated some symbols (like structs) with __declspec(dllexport). If it weren't for libtool's paranoia about passing flags to the linker, you'd just say '--export-all' and be done. But libtool won't let you do that, so life is hard. This is the glib case.

If you want #1, you just let ld do the work. No need for export_symbols or any of that junk.

If you want #2, the *Developer* should specify that in the makefile. E.g. libfoo_la_LDFLAGS=.... --exclude-symbols SYMBOL,SYMBOL,... or --exclude-libs LIB,LIB,... (to avoid exporting symbols pulled in from static convenience libraries). But that's a *developer* thing; libtool shouldn't attempt to do that itself. It isn't clairvoyant. Perhaps the developer should insure that the Makefile only puts --exclude-symbols ... when building for cygwin/mingw by using conditionals. But it's a developer thing.

Now, if on unix, a developer used --export-symbols or --export-symbols-regex as a means of NOT exposing certain symbols to clients, then we need to take special care when porting to windows. Create a .def file and add it (conditionally) to the _SOURCES [BUZZ! doesn't work], or add --exclude-[lib|symbol] to _LDFLAGS (conditionally) [BUZZ! Doesn't work. Damn, but libtool is picky about the flags it passes to the linker], etc. But this should be RARE.

If you want #3, then the *developer* should specify a .def file as one of the library's dependencies (perhaps conditional on cygwin/mingw). End of story.

And --export-symbols/--expoprt-symbols-regex should continue to be handled on cygwin exactly as it is handled on other unixes: strip or don't strip. It shouldn't affect the "windowsish" exporting process. That's the job of --exclude-[lib|symbobs] and .def file, if such functionality is needed.

But apparently Tor Lilqvist disagrees:

Now, having said all of that, it appears that this is EXACTLY what the glib folks have done. They explicitly add "-export-symbols glib.def" IF and ONLY IF OS_WIN32 (eg. not for cygwin, which has PLATFORM_WIN32 true, but OS_WIN32 false). IMO, that is a *misunderstanding* of what --export-symbols does in CURRENT libtool, even if it was correct for OLD libtool -- but Tor Lilqvist [the gtk on windows dude] uses a forked version of libtool anyway, so...

As it happens, they've managed to outsmart the auto-export code when they try this OLD libtool usage with a NEW libtool program. Auto-export ONLY happens if BOTH of the following are true
1) there is no .def file
2) there are no explicit __declspec(dllexport) decorations on any symbols.

1) is true, because the .def file is only used for OS_WIN32 (not cygwin). However, 2) is false, because there are five symbols that are explicitly decorated:
g_io_watch_funcs in glib/giowin32.c
g_timeout_funcs in glib/gmain.c
g_idle_funcs in glib/gmain.c
g_ascii_table in glib/gstrfuncs.c
g_thread_functions_for_glib_use in glib/gthread.c

These are the only DATA exports from glib that are "complex" (see 'info ld' and search on 'auto-import').

So, the "right thing" on cygwin is one of these three things:

include the .def file -- but as explained exhaustively above, you can't do it the way Tor does, the (new) "real" libtool doesn't use --export-symbols the way he thinks (although the old libtool did). So, you'd have to conditionally include it as a part of glib_la_SOURCES. But THAT doesn't work EITHER -- because libtool says "what's this random file doing in my argument list, it's not a .lo, it's not .la, throw it out" So, Tor's solution is the only one that can work -- but it requires that $archive_expsyms_cmd be fixed up etc etc... (*)

or

add --export-all-symbols as one of the glib_la_LDFLAGS switches (this will re-enable the auto-export feature of ld, even though SOME symbols are explicitly decorated). However, libtool does not pass "--export-all-symbols" thru to the linker, so this might require a patch to libtool. Probably not the best way to do this.

or

remove the explicit decorations on those five variables, and rely on the runtime-pseudo-relocs feature of binutils. This won't work until cygwin-1.3.18 is released; it's a very new feature. And --enable-runtime-pseudo-relocs is not the default for ld yet, so you need to add that flag to LDFLAGS...but libtool refuses to pass that switch to the linker! So this won't work until (a) the new cygwin kernel comes out, and (b) --enable-runtime-pseudo-relocs becomes the default on cygwin's ld. (b) won't happen for quite some time, yet.

-----

So, it appears that that the only way to address this in the short term is...restore some form of the old libtool-1.4.x code for $archive_expsym_cmd.

Fixup the official libtool so that IF '--export-symbols <FILENAME>' is specified, then on cygwin/mingw assume that the file contains a .def-format specification, and $archive_expsyms_cmd includes it as "just one of the object files" and doesn't use the --retain-symbols-file at all. (On other platforms, go ahead and use --retain-symbols-file, etc). This means that it would be up to the developer to do the following, IF he wants to restrict the list of export symbols on both unix and windows:
1) provide foo.exp which is just a list of symbols
2) provide foo.def which is a def-format list of symbols, with DATA specifiers, etc
3) conditionally add "--export-symbols foo.exp" on unix, and "--export-symbols foo.def" on windows.

This would require all of the changes to $export_symbols_cmds described above, too, so that libtool's --export-symbols-regex switch could still work (currently, that is hopelessly broken on cygwin).

Now that I've thought about it, I believe that this IS what should happen in the libtool codebase. It seems that some of the "old stuff" that was kept around for dlltool and was recently removed, should be retained so that the new-and-improved, use-auto-import-as-much-as-possible libtool can properly handle corner cases like these.

But, that's a separate issue from "how to get glib working on cygwin". IF all of those changes were in the current libtool, then yeah -- do it that way. But they are not (yet). The long-term solution is to remove the __declspec decorators on those five symbols when __CYGWIN__, and rely on --enable-runtime-pseudo-relocs once it becomes default, and on the latest cygwin kernel which should be out in a few days. (If you wanted to try this method out BEFORE --enable-runtime-pseudo-relocs becomes the default, the only way to pass that flag thru libtool appears to be create a shell script and pretend that it is your compiler:

--gcc--
/path/to/real/gcc -Wl,--enable-runtime-pseudo-relocs $*
-------

Because libtool is REALLY picky about what it passes thru to the linker. Sheesh!

Eventually pseudo-relocs will be the default on cygwin -- and maybe mingw -- but not for a number of months, I'm sure.)

Anyway, thanks for bringing up this issue; it appears that the current libtool is not quite ready for absolutely every possible use that people want...Looks like there are a few things I need to do over Christmas break... :-)

--Chuck


--
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Bug reporting: http://cygwin.com/bugs.html
Documentation: http://cygwin.com/docs.html
FAQ: http://cygwin.com/faq/


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]