Re: Difficulties while adding local init_regs support to libebl

[Ben Gamari <ben at smart-cactus dot org>]

Mark Wielaard <mjw@redhat.com> writes:

> Hi Ben,
>
> On Fri, Sep 04, 2015 at 01:29:18PM +0200, Ben Gamari wrote:
>> >> At this point I'm a bit unsure of how to proceed. I can think of a few
>> >> options,
>> >> 
>> >>   * teach dwfl_linux_local_attach to report libebl when it gets
>> >>     loaded. This would be rather ugly as we would either need to
>> >>     re-read the /proc map data or somehow extract the necessary
>> >>     information from the dynamic linker
>> >
>> > It might be interesting to have a Dwfl *dwfl_linux_local_report () that
>> > creates a Dwfl using the dynamic linker structures. It could come with a
>> > _refresh function or keep things automagically up to date (although that
>> > might be tricky with threading). But that seems a but more work than
>> > what you signed up for right now.
>> >
>> That might be the best option although I'm not entirely sure what
>> (reasonably portable) interfaces exist for this task.
>
> There is dl_iterate_phdr which is a GNU extensions, but some other
> systems have something similar.
>
Right, I actually went ahead and implemented your suggestion after
stumbling upon this in the manpages.

>> Unfortunately it seems that libebl_x86_64.so doesn't even appear in
>> /proc/$PID/maps, so dwfl_linux_proc_report doesn't find it. I have no
>> idea why considering that gdb knows about it. Perhaps the reason is that
>> it is loaded with dlopen?
>
> Yes, it will only show up after it is dlopened. And not anymore after
> it is dlclosed. Which is why you want to load the backend (and not
> unload it again) before listing the current modules.
>
>> This makes me think that your suggestion of building up the module list
>> From the dynamic linker's structures might be necessary.
>
> Although it wouldn't bad to have such an interface, I don't think it is
> necessary. See below.
>
Too late; already implemented ;).

>
> Passing NULL is kind of lazy. An argument could be made that having state
> attached to a Dwfl is useful on itself, even without any Dwfl_Modules
> registered. For example one might just be interested in the threads and
> the current pc address for a quick sample/profile without even wanting
> to extract symbols or a backtrace. And like in your case you might want
> to attach state to a Dwfl and then later refresh the Dwfl_Modules to
> accurately model the current state of loaded libraries.
>
> Attached is a quick patch to implement that for dwfl_linux_proc_attach.
> I think you could use the same for dwfl_linux_local_attach to make the
> above possible.
>
Ahh, I see. Indeed this seems to fix it. Will this patch be making it upstream?


Unfortunately, there is one more issue that took me far too long to
realize. The trouble is the indirection that libebl calls experience
interact poorly with collection of the initial registers.

Consider the call stack that you would see when collecting initial
registers,

    #0  x86_64_set_initial_registers_local () at x86_64_initreg_local.s:18
    #1  0x00007ffff7bca385 in ebl_set_initial_registers_local (ebl=0x6020a0, setfunc=0x7ffff7bc3e56 <local_thread_state_registers_cb>, arg=0x7fffffffdf50) at eblinitreg.c:55
    #2  0x00007ffff7bc3f5c in local_set_initial_registers (thread=0x7fffffffdf50, thread_arg=0x0) at linux-local-attach.c:98
    #3  0x00007ffff7bbff2d in dwfl_thread_getframes (thread=0x7fffffffdf50, callback=0x400c46 <frame_cb>, arg=0x602010) at dwfl_frame.c:428
    #4  0x00007ffff7bbfe2c in get_one_thread_frames_cb (thread=0x7fffffffdf50, arg=0x7fffffffdfd0) at dwfl_frame.c:392
    #5  0x00007ffff7bbfd40 in getthread (dwfl=0x602010, tid=27815, callback=0x7ffff7bbfdf6 <get_one_thread_frames_cb>, arg=0x7fffffffdfd0) at dwfl_frame.c:357
    #6  0x00007ffff7bbfe72 in dwfl_getthread_frames (dwfl=0x602010, tid=27815, callback=0x400c46 <frame_cb>, arg=0x602010) at dwfl_frame.c:401
    #7  0x0000000000400ece in dump_backtrace () at backtrace-local.c:106
    #8  0x0000000000400f32 in main (argc=1, argv=0x7fffffffe108) at backtrace-local.c:122

Where x86_64_set_initial_registers_local is in libebl and is responsible for collecting
the initial register values and invoking an ebl_tid_registers_t callback
with them. More specifically, this function records precisely the
register values that it was entered with.

Unfortunately, this is idea is dead on arrival (which I didn't realize
until far too much head-scratching): the stack frames between
x86_64_set_initial_registers_local and dwfl_thread_getframes are gone by
the time we try to unwind. We return three times back to
dwfl_thread_getframes, before calling into the unwinder.

This didn't affect me in GHC since there were no stack frames sitting
between the initial register collection and the unwinder.

I see two ways to work around this,

  1. Implement a small amount of unwinding in the initial register collection
     code to get back to a stack frame which won't disappear from
     beneath our feet

  2. Ensure that there are no stack frames sitting between initial
     register collection and the unwinder

Option 1 strikes me as quite complex and fragile. Unfortunately the only
way to implement Option 2 would be to implement a special
`dwfl_local_getframes` interface, which would be specialized to the
current architecture. Neither option is particularly appealing.

What do you think?

Cheers,

- Ben

Attachment: signature.asc
Description: PGP signature