This is the mail archive of the
elfutils-devel@sourceware.org
mailing list for the elfutils project.
Re: dwfl_module_getsym vs. prelink
- From: Mark Wielaard <mjw at redhat dot com>
- To: elfutils-devel at lists dot fedorahosted dot org
- Date: Fri, 15 Nov 2013 14:35:00 +0100
- Subject: Re: dwfl_module_getsym vs. prelink
On Thu, 2013-11-14 at 16:00 -0800, Josh Stone wrote:
> On IRC, I mentioned to Mark some bias issues I was having with values
> from dwfl_module_getsym, and I promised a smaller reproducer than using
> systemtap, so here's my attempt to recreate it with test code that
> already exists in elfutils.
Thanks for this.
Just for reference, what dwfl_module_getsymtab()/dwfl_module_getsym ()
gets you is the symbol table and symbols for the Dwfl_Module. This can
either be the full symtab from the either the main elf file or the
separate debug file. Or the minimal .dynsym table (always from the main
file), possibly augmented by the minisymtab table from the aux file (the
compressed .gnu_debugdata embedded elf file). In all cases the symbol
table is always presented as a normal symtab table (first zero entry,
locals before globals). And for symbols referring to allocated sections
the st_value is adjusted to the Dwfl_Module address range inside the
Dwfl. This makes it easy to get relative addresses for such symbols
through dwfl_module_relocate_address() or to find the symbol associated
with a particular address in a module with dwfl_module_addrsym().
prelinking makes some if this slightly tricky, and maybe you found a
bug, we will see below. A shared library (ET_DYN) that is prelinked has
its p_vaddr set (see eu-readelf -l) just like a normal executable
(ET_EXEC), as a hint to the shared library loader where to load the .so
in memory so no relocations are necessary. Which also means the symbol
values that refer to allocated sections don't have to be adjusted for
that load address. But prelinking occurs on on the main elf file only,
the separate debug or aux file might be "out of sync", so symbols coming
from different files might need different adjustment to match the Dwfl
load address of the Dwfl_Module they come from.
Note that we are currently having a discussion precisely about these
functions because although the symbols might come from different files
and the symbols do refer to a particular section, the user cannot easily
determine which Elf file (main, debug or aux) they really came from
(a .debug file as more sections which might be differently ordered from
the main elf file). Especially since the full .symtab might be inside
either the main elf or the separate .debug file. So we are debating
whether to drop the whole section reference from the symbol or adding a
reference to the Elf file (and bias).
> The problem I'm having surrounds gnu_debugdata, and tests/dwflsyms
> already covers this scenario with testfilebazmdb and testfilebazmin. So
> let's experiment with prelinking these.
>
> Note, I extracted all testfilebaz* in my working directory, mainly to
> make sure the .debug was also available. I'm also grepping output to
> reduce how long this email gets. :)
Lets make the email a little longer :) Adding and comparing with the
p_vaddr (eu-readelf -l) and original symbol tables (eu-readelf -s
[--elf-section]).
You are using tests/dwflsyms which simply a dwfl_module_getsymtab() and
iterates over all symbols with dwfl_module_getsym() and prints the
symbol values returned. It then also asserts that for function symbols
dwfl_module_addrsym() for the given st_value provides the same symbol.
Lets extend it a little so we also print out the relative address:
diff --git a/tests/dwflsyms.c b/tests/dwflsyms.c
index cae3fbe..55f2653 100644
--- a/tests/dwflsyms.c
+++ b/tests/dwflsyms.c
@@ -83,7 +83,7 @@ list_syms (struct Dwfl_Module *mod,
GElf_Sym sym;
GElf_Word shndxp;
const char *name = dwfl_module_getsym (mod, ndx, &sym, &shndxp);
- printf("%4d: %s\t%s\t%s (%" PRIu64 ") %#" PRIx64 "\n",
+ printf("%4d: %s\t%s\t%s (%" PRIu64 ") %#" PRIx64,
ndx, gelf_type (&sym), gelf_bind (&sym), name,
sym.st_size, sym.st_value);
@@ -97,7 +97,12 @@ list_syms (struct Dwfl_Module *mod,
GElf_Word ashndxp;
const char *aname = dwfl_module_addrsym (mod, addr, &asym, &ashndxp);
assert (strcmp (name, aname) == 0);
+
+ int res = dwfl_module_relocate_address (mod, &addr);
+ assert (res != -1);
+ printf(", rel: %#" PRIx64 "", addr);
}
+ printf ("\n");
}
return DWARF_CB_OK;
> First testfilebazmdb (dynsym + gnu_debugdata + .debug):
>
> > $ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
> > 45: FUNC LOCAL foo (20) 0x814
> > 58: FUNC GLOBAL bar (44) 0x828
> > 70: FUNC GLOBAL main (35) 0x7f0
This one of the simpler situations, we get all symbols from the
full .symtab and both have the same load address. All values already are
relative (zero based).
$ eu-readelf -l testfilebazmdb | grep LOAD | head -1
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -l testfilebazdbg.debug | grep LOAD | head -1
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazdbg.debug | grep -w FUNC | grep -wE 'main|foo|bar'
45: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
58: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
70: 00000000000007f0 35 FUNC GLOBAL DEFAULT 13 main
$ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
45: FUNC LOCAL foo (20) 0x814, rel: 0x814
58: FUNC GLOBAL bar (44) 0x828, rel: 0x828
70: FUNC GLOBAL main (35) 0x7f0, rel: 0x7f0
> > $ /usr/sbin/prelink -N testfilebazmdb
> > $ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
> > 45: FUNC LOCAL foo (20) 0x3005800814
> > 58: FUNC GLOBAL bar (44) 0x3005800828
> > 70: FUNC GLOBAL main (35) 0x30058007f0
$ eu-readelf -l testfilebazmdb | grep LOAD | head -1
LOAD 0x000000 0x0000003000000000 0x0000003000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -l testfilebazdbg.debug | grep LOAD | head -1
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazdbg.debug | grep -w FUNC | grep -wE 'main|foo|bar'
45: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
58: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
70: 00000000000007f0 35 FUNC GLOBAL DEFAULT 13 main
$ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
45: FUNC LOCAL foo (20) 0x3000000814, rel: 0x814
58: FUNC GLOBAL bar (44) 0x3000000828, rel: 0x828
70: FUNC GLOBAL main (35) 0x30000007f0, rel: 0x7f0
Relative addresses and st_values do make sense.
> > $ /usr/sbin/prelink -u testfilebazmdb
> > $ /usr/sbin/prelink -r 0x70000000 testfilebazmdb
> > $ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
> > 45: FUNC LOCAL foo (20) 0x70000814
> > 58: FUNC GLOBAL bar (44) 0x70000828
> > 70: FUNC GLOBAL main (35) 0x700007f0
>
> This all looks fine; all three functions move around together.
$ eu-readelf -l testfilebazmdb | grep LOAD | head -1
LOAD 0x000000 0x0000000070000000 0x0000000070000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -l testfilebazdbg.debug | grep LOAD | head -1
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazdbg.debug | grep -w FUNC | grep -wE 'main|foo|bar'
45: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
58: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
70: 00000000000007f0 35 FUNC GLOBAL DEFAULT 13 main
$ ./dwflsyms -e testfilebazmdb | grep -w FUNC | grep -wE 'main|foo|bar'
45: FUNC LOCAL foo (20) 0x70000814, rel: 0x814
58: FUNC GLOBAL bar (44) 0x70000828, rel: 0x828
70: FUNC GLOBAL main (35) 0x700007f0, rel: 0x7f0
And again the relative addresses look fine.
> Now testfilebazmin (dynsym + gnu_debugdata)
>
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 6: FUNC LOCAL foo (18) 0x400498
> > 42: FUNC GLOBAL main (35) 0x7f0
> > 45: FUNC GLOBAL bar (44) 0x4004aa
>
> Already a bad start - foo/bar look like symbols that would belong in
> ET_EXEC (which indeed the embedded gnu_debugdata really is). I would
> not expect that to show up in the main file getsym list, even though
> run-dwflsyms.sh is asserting this output.
>
> > $ /usr/sbin/prelink -N testfilebazmin
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 12: FUNC GLOBAL main (35) 0x30058007f0
>
> Here we completely lost foo/bar!
>
> > $ /usr/sbin/prelink -u testfilebazmin
> > $ /usr/sbin/prelink -r 0x70000000 testfilebazmin
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 6: FUNC LOCAL foo (18) 0x400498
> > 42: FUNC GLOBAL main (35) 0x700007f0
> > 45: FUNC GLOBAL bar (44) 0x4004aa
>
> Ok, foo/bar came back but still refuse to relocate with main.
>
>
> I would expect testfilebazmdb and testfilebazmin to behave the same way.
> Even if the latter's symbol tables are less complete, the symbols that
> are presented ought to be consistent in their base offset, so any can be
> used with dwfl_module_relocate_address.
>
>
> The ET_EXEC gnu_debugdata in both bazmdb and bazmin is suspicious. I
> followed your directions to generate all these testfilebaz* on Fedora
> 19, and I got ET_DYN gnu_debugdata. The symbols are also the same size
> and location in my bazmdb and bazmin, whereas you can see above the test
> files in elfutils.git are not.
Indeed. This is embarrassing. It looks like the test files as checked
into git (plus the expected output!) are just bad. It is curious how
that happened because following the instructions to recreate these files
in tests/run-readelf-s.sh does produce a correctly embedded
gnu_debugdata. I'll fix this and double check the expected output.
Thanks for catching this and sorry for the mess :{
> Repeating my prelink sequence on these new binaries, bazmdb still looks
> good, and now bazmin looks like this:
>
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 8: FUNC LOCAL foo (20) 0x75c
> > 47: FUNC GLOBAL bar (40) 0x770
> > 51: FUNC GLOBAL main (35) 0x738
$ eu-readelf -l testfilebazmin | grep LOAD | head -1 LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
12: 00000000000007f0 35 FUNC GLOBAL DEFAULT 13 main
$ eu-readelf --elf-section -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
7: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
36: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
$ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
8: FUNC LOCAL foo (20) 0x814, rel: 0x814
47: FUNC GLOBAL main (35) 0x7f0, rel: 0x7f0
49: FUNC GLOBAL bar (44) 0x828, rel: 0x828
So indeed the corrected one does look good.
> > $ /usr/sbin/prelink -N testfilebazmin
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 8: FUNC LOCAL foo (20) 0x300580075c
> > 47: FUNC GLOBAL bar (40) 0x3005800770
> > 51: FUNC GLOBAL main (35) 0x3005800738
$ eu-readelf -l testfilebazmin | grep LOAD | head -1
LOAD 0x000000 0x0000003000000000 0x0000003000000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
12: 00000030000007f0 35 FUNC GLOBAL DEFAULT 13 main
$ eu-readelf --elf-section -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
7: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
36: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
$ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
8: FUNC LOCAL foo (20) 0x3000000814, rel: 0x814
47: FUNC GLOBAL main (35) 0x30000007f0, rel: 0x7f0
49: FUNC GLOBAL bar (44) 0x3000000828, rel: 0x828
That indeed looks also as expected, the aux symbols values are adjusted
to the new base address.
> > $ /usr/sbin/prelink -u testfilebazmin
> > $ /usr/sbin/prelink -r 0x70000000 testfilebazmin
> > $ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
> > 8: FUNC LOCAL foo (20) 0x75c
> > 47: FUNC GLOBAL bar (40) 0x770
> > 51: FUNC GLOBAL main (35) 0x738
>
> Normal prelink is better here, but now prelink -r didn't actually appear
> to move any of foo, bar, or main!?!
$ eu-readelf -l testfilebazmin | grep LOAD | head -1 LOAD 0x000000 0x0000000070000000 0x0000000070000000 0x000a5c 0x000a5c R E 0x200000
$ eu-readelf -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
12: 00000000700007f0 35 FUNC GLOBAL DEFAULT 13 main
$ eu-readelf --elf-section -s testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
7: 0000000000000814 20 FUNC LOCAL DEFAULT 13 foo
36: 0000000000000828 44 FUNC GLOBAL DEFAULT 13 bar
$ ./dwflsyms -e testfilebazmin | grep -w FUNC | grep -wE 'main|foo|bar'
8: FUNC LOCAL foo (20) 0x814, rel: 0xffffffff90000814
47: FUNC GLOBAL main (35) 0x700007f0, rel: 0x7f0
49: FUNC GLOBAL bar (44) 0x828, rel: 0xffffffff90000828
OK, that is definitely a bug. But I cannot immediately explain why it
worked fine in the -N case and not in the -r case. Except for the
specific load address there should be no difference.
> However, this grep is too tight,
> because some a few symbols did report moved, like _end. I'll end with
> that full output:
>
> > $ ./dwflsyms -e testfilebazmin
> > 0: NOTYPE LOCAL (0) 0
> > 1: SECTION LOCAL (0) 0x70000238
> > 2: FUNC LOCAL deregister_tm_clones (0) 0x650
> > 3: FUNC LOCAL register_tm_clones (0) 0x680
> > 4: FUNC LOCAL __do_global_dtors_aux (0) 0x6c0
> > 5: OBJECT LOCAL __do_global_dtors_aux_fini_array_entry (0) 0x200df0
> > 6: FUNC LOCAL frame_dummy (0) 0x700
> > 7: OBJECT LOCAL __frame_dummy_init_array_entry (0) 0x200de8
> > 8: FUNC LOCAL foo (20) 0x75c
> > 9: NOTYPE LOCAL __init_array_end (0) 0x200df0
> > 10: NOTYPE LOCAL __init_array_start (0) 0x200de8
> > 11: SECTION LOCAL (0) 0x238
> > 12: SECTION LOCAL (0) 0x254
> > 13: SECTION LOCAL (0) 0x274
> > 14: SECTION LOCAL (0) 0x298
> > 15: SECTION LOCAL (0) 0x2c8
> > 16: SECTION LOCAL (0) 0x3d0
> > 17: SECTION LOCAL (0) 0x47a
> > 18: SECTION LOCAL (0) 0x490
> > 19: SECTION LOCAL (0) 0x4b0
> > 20: SECTION LOCAL (0) 0x570
> > 21: SECTION LOCAL (0) 0x5b8
> > 22: SECTION LOCAL (0) 0x5e0
> > 23: SECTION LOCAL (0) 0x620
> > 24: SECTION LOCAL (0) 0x814
> > 25: SECTION LOCAL (0) 0x820
> > 26: SECTION LOCAL (0) 0x824
> > 27: SECTION LOCAL (0) 0x868
> > 28: SECTION LOCAL (0) 0x200de8
> > 29: SECTION LOCAL (0) 0x200df0
> > 30: SECTION LOCAL (0) 0x200df8
> > 31: SECTION LOCAL (0) 0x200e00
> > 32: SECTION LOCAL (0) 0x200e08
> > 33: SECTION LOCAL (0) 0x200fd8
> > 34: SECTION LOCAL (0) 0x201000
> > 35: SECTION LOCAL (0) 0x201030
> > 36: SECTION LOCAL (0) 0x20103c
> > 37: NOTYPE WEAK _ITM_deregisterTMCloneTable (0) 0
> > 38: FUNC GLOBAL __libc_start_main (0) 0
> > 39: NOTYPE WEAK __gmon_start__ (0) 0
> > 40: NOTYPE WEAK _Jv_RegisterClasses (0) 0
> > 41: NOTYPE WEAK _ITM_registerTMCloneTable (0) 0
> > 42: FUNC WEAK __cxa_finalize (0) 0
> > 43: NOTYPE GLOBAL _edata (0) 0x7020103c
> > 44: NOTYPE GLOBAL _end (0) 0x70201040
> > 45: NOTYPE GLOBAL __bss_start (0) 0x7020103c
> > 46: FUNC GLOBAL __libc_csu_fini (2) 0x810
> > 47: FUNC GLOBAL bar (40) 0x770
> > 48: FUNC GLOBAL _fini (0) 0x814
> > 49: FUNC GLOBAL __libc_csu_init (101) 0x7a0
> > 50: FUNC GLOBAL _start (0) 0x620
> > 51: FUNC GLOBAL main (35) 0x738
> > 52: FUNC GLOBAL _init (0) 0x5b8
It looks like those came directly from the .dynsym (and so didn't need
any adjustment):
$ eu-readelf -s testfilebazmin
Symbol table [ 5] '.dynsym' contains 14 entries:
2 local symbols String table: [ 6] '.dynstr'
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UNDEF
1: 0000000070000238 0 SECTION LOCAL DEFAULT 1
2: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF _ITM_deregisterTMCloneTable
3: 0000000000000000 0 FUNC GLOBAL DEFAULT UNDEF __libc_start_main(a)GLIBC_2.2.5 (2)
4: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF __gmon_start__
5: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF _Jv_RegisterClasses
6: 0000000000000000 0 NOTYPE WEAK DEFAULT UNDEF _ITM_registerTMCloneTable
7: 0000000000000000 0 FUNC WEAK DEFAULT UNDEF __cxa_finalize(a)GLIBC_2.2.5 (2)
8: 000000007020103c 0 NOTYPE GLOBAL DEFAULT 25 _edata
9: 0000000070201040 0 NOTYPE GLOBAL DEFAULT 26 _end
10: 0000000070000860 137 FUNC GLOBAL DEFAULT 13 __libc_csu_init
11: 000000007020103c 0 NOTYPE GLOBAL DEFAULT 26 __bss_start
12: 00000000700007f0 35 FUNC GLOBAL DEFAULT 13 main
13: 00000000700008f0 2 FUNC GLOBAL DEFAULT 13 __libc_csu_fini
I'll try and find the bug.
Thanks,
Mark