Re: The gdb x86 function prologue parser

[Mark Kettenis <mark dot kettenis at xs4all dot nl>]

   From: Jason Molenda <jmolenda@apple.com>
   Date: Tue, 7 Jun 2005 22:51:36 -0700

Hi Jason,

Sorry I didn't reply before; I still was on vacation when you sent out
this mail.

   As was announced yesterday, we'll be transitioning from PPC to x86  
   over the next couple of years.  Over the last few months I had a  
   delightful little introduction to the x86 architecture (read: read  
   the IA32 PDFs until I couldn't see straight any more) and worked to  
   get our x86 gdb support up to snuff for this software release.

Seems like a step backwards to me, but hey who am I to judge Apple.
Watched the QuickTime video where your big boss announces the thingy.
Seems he's firmly in bed with Intel and not AMD.  Still makes me
wonder if Apple is going to provide a 64-bit version of MacOS X for
the new platform.

Another question that I have is what calling convention MacOS X will
use.  Is it something like the System V ABI where the caller is
supposed to clean up the stack after a function call?  I certainly
hope so since the Microsoft way of doing things poses a major headache
for prologue scanning.

   One of the biggest problems we found was the x86 function prologue  
   parser is remarkably weak.  We have a very mature and featureful  
   prologue parser on our ppc side and an amazing number of bugs were  
   directed my way as we had people pounding on the x86 side.  We aren't  
   using DWARF yet, so CFI can't save our bacon -- the prologue parser  
   has to work or our gdb fails.

I don't know to what extent your version of gdb is synched with the
FSF tree, but if it is anything close to gdb 6.x, then yes, you're
pretty much hosed if the prologue scanner fails.  It's no surprise to
me though that the prologue scanner appears a bit weak to you though.
It started out as a prologue scanner for the origional System V
compiler with some additions to older GCC versions when I took over.
At that point, GCC still had a fixed prologue.  When GCC 3.x started
scheduling prologue instructions, it also started generating usable
DWARF CFI, so whe took the conscious decision to rely on that and only
improve the prologue scanner on an as-needed basis.  Since GCC 3.x
targets for all major free OS'es all use DWARF by default, this means
that the prologue scanner really only handles some bits of
hand-optimized assembler.

   There are a couple classes of changes I made, and I spent today  
   trying to massage them into some kind of presentable form.  This is  
   not perfect -- well, to be honest, this is just a first sketch -- but  
   this is a HUGE improvement over the existing facilities.  I wrote the  
   code while under immense deadline pressure so I'm not particularly  
   interested in how I implemented any of it.  But changes akin to these  
   are necessary if you want to debug programs with optimized code on  
   the stack and without CFI.  I'll guarantee it.

It'd be great if we could improve the prologue scanner.  This reminds
me that I really should start testing -gstabs.

   Roughly speaking, here are the class of changes included in this patch.

   1. i386_match_insn() bug fixes.  It wouldn't work for an instruction  
   pattern of 1 byte, and it would never check anything beyond the 2nd  
   byte (notice where the final "return insn" is located).  I've added  
   patterns that can match prologue instructions, so exceptions had to  
   be added for the big two (push %ebp, mov %esp, %ebp) and the  
   equivalent ones used in the first frame (_start()).

Duh.  I must have been on drugs when I wrote that code.

   2. i386_frame_setup_skip_insns table expansion.  Because you can't  
   skip over an unknown instruction on x86 without knowing its length,  
   this was of paramount importance.  Initially I waited for users to  
   tell me of prologues that gdb was failing on, but this was taking too  
   long and there were too many instructions scheduled into prologues  
   for me to hear of them in time.  So I wrote a little maintenance  
   command (not included in the patch to keep things simple) which would  
   tell you if gdb could parse through the prologue of a given  
   function.  Then with a couple of shell scripts, I could have gdb try  
   to analyze the prologues of every function in every library on my  
   MacOS X system and show me the ones it failed on.  I'd add them to  
   this list.  I also made a little testsuite generator where the input  
   looks like

   # SOURCE: RedHat FedoraCore2 /lib/ld-2.3.3.so _dl_reloc_bad_type
   # PROLOGUE
      push %ebp
      shl $0x5, %ecx # [ 0xc1 0xe1 0x05 ]
      mov %esp, %ebp
      sub $0x8, %esp
   # EPILOGUE
      add $0x8, %esp
      pop %ebp
      ret

   and a script that transforms the patterns into a test program and a  
   Dejagnu expect script.  So you can ensure that you don't regress the  
   prologue parser.  This was the lesson we learned in writing our PPC  
   parser -- we have this wonderfully ornate parser with lots of  
   exceptions and known tricks, but no testsuite for it.  So whenever we  
   change it we're cringing because the gdb testsuite has nothing useful  
   in it.  (you need optimized, no debug info test cases to be sure it's  
   still working right).  The testsuite stuff isn't included in this  
   patch, but I'll put that together soon and send it along if anyone's  
   interested.

Certainly, if we change the prologue scanner to deal with new patterns
we should test these patterns in the testsuite.

   3. relatively minor changes to i386_analyze_frame_setup().  It had to  
   have the push %ebp as the very first instruction or it would give  
   up.  That's really bad -- the compiler can (and does) schedule all  
   sorts of stuff before that instruction.

I believe you.  I'm wondering though if the current way the prologue
scanner is built up makes sense for this new world of completely
scheduled prologues.

   4. new function, i386_find_esp_adjustments().  This is used in a  
   frameless leaf function where the compiler may create space on the  
   stack for local variables and stuff, but doesn't call anything so it  
   doesn't save the caller's frame pointer.  And it allows -fomit-leaf- 
   frame-pointer codegen to be debugged.  -fomit-frame-pointer is a  
   whole lot more complicated, but this wasn't so bad.  (we didn't end  
   up enabling -fomit-leaf-frame-pointer in this release because of the  
   schedule time constraints, but that's why I wrote it)

Being able to debug -fomit-frame-pointer code without CFI probably
means that instead of scanning the prologue, we'll have to scan the
complete function up to the current instruction pointer.  I really
wonder if that's the way we should go.

   5. Huge i386_frame_cache() changes.  There's no way around it, this  
   function is just not right.  It doesn't handle frameless functions  
   correctly at all.  It's written without a clear understanding of the  
   different classes of functions it needs to handle and works primarily  
   by luck.  And for goodness sakes, if we can't figure out anything  
   about a function that's not at the top of the stack, don't you think  
   it'd be reasonable to assume that the function has set up a stack  
   frame and saved the caller's EBP?  Sure seems like a reasonable  
   assumption to me.  Why can't this function do something even that  
   basic?  This function really cheesed my mellow.

Well, it handles most of the frameless functions encountered on a
GNU/Linux system with GCC 3.2 fine.  And no, assuming that a function
has set up a stack frame isn't right; it makes gdb silently skip
function calls in backtraces.  That can be very confusing.  As I've
stated before, I'd rather have a backtrace that's obviously wrong than
one that silently omits things.  But it's a trade-off.  Maybe
improving the prologue scanner can shift the balance far enough that
the assumption that a stack frame has been set up makes more sense
again.

Assuming that a function saves the previous value of %ebp is demanded
by the System V ABI, but GCC might violate the ABI for static
functions where it knows the caller has already saved %ebp.

   Mark, I want to say that I'm not directing any of these criticisms  
   towards you -- I've been looking over the changes you've made and  
   they're definite improvements over the existing code.  The existing  
   code bites, though.  I can't even begin to imagine how annoyed  
   developers using the FSF gdb on x86 must be.  The changes I'm sending  
   here are not a panacea/beautiful/perfect, but *functionally* they're  
   a huge improvement.  Now that our release is out there I'll be more  
   than happy to revisit the decisions/implementation that I came up  
   with on little sleep. :-)

Great.  I haven't looked at your patch in detail yet.  But it sounds
like some of the improvements can be made right away, so let's get
working on this ;-).

   Oh, and I ran my "find all prologues gdb can't parse" on a FedoraCore  
   2 system I have handy here at the office today and added the patterns  
   of the biggest offenders.  There are still a few patterns I need to  
   add to get 100% parsing success but I want to go home :-) so that'll  
   be for another day.

   We're in the middle of an all-week Apple developer's conference, so  
   my replies may not be very speedy (I'm off-line 10-12 hours a day; I  
   slipped away to get this patch together today) until the weekend.   
   But I'll try to stay on top of any questions or comments and address  
   them promptly.

No problem; I was having a vacation anyway ;-).

Mark