[binutils-gdb] Fix displaced-stepping RIP-relative VEX-encoded instructions (AVX) (PR gdb/22499)

[sergiodj+buildbot at sergiodj dot net]

*** TEST RESULTS FOR COMMIT 50a1fdd59c1777672a9be0e81fe2301c2a115fce ***

Author: Pedro Alves <palves@redhat.com>
Branch: master
Commit: 50a1fdd59c1777672a9be0e81fe2301c2a115fce

Fix displaced-stepping RIP-relative VEX-encoded instructions (AVX) (PR gdb/22499)

PR gdb/22499 is about a latent bug exposed by the switch to "maint set
target-non-stop on" by default on x86-64 GNU/Linux, a while ago.  With
that on, GDB is also preferring to use displaced-stepping by default.

The testcase in the bug is failing because GDB ends up incorrectly
displaced-stepping over a RIP-relative VEX-encoded instruction, like
this:

 0x00000000004007f5 <+15>:    c5 fb 10 05 8b 01 00 00 vmovsd 0x18b(%rip),%xmm0        # 0x400988

While RIP-relative instructions need adjustment when relocated to the
scratch pad, GDB ends up just copying VEX-encoded instructions to the
scratch pad unmodified, with the end result that the inferior ends up
executing an instruction that fetches/writes memory from the wrong
address...

This patch teaches GDB about the VEX-encoding prefixes, fixing the
problem, and adds a testcase that fails without the GDB fix.

I think we may need a similar treatment for EVEX-encoded instructions,
but I didn't address that simply because I couldn't find any
EVEX-encoded RIP-relative instruction in the gas testsuite.  In any
case, this commit is forward progress as-is already.

gdb/ChangeLog:
2017-12-04  Pedro Alves  <palves@redhat.com>

	PR gdb/22499
	* amd64-tdep.c (amd64_insn::rex_offset): Rename to...
	(amd64_insn::enc_prefix_offset): ... this, and tweak comment.
	(vex2_prefix_p, vex3_prefix_p): New functions.
	(amd64_get_insn_details): Adjust to rename.  Also skip VEX2 and
	VEX3 prefixes.
	(fixup_riprel): Set VEX3.!B.

gdb/testsuite/ChangeLog:
2017-12-04  Pedro Alves  <palves@redhat.com>

	PR gdb/22499
	* gdb.arch/amd64-disp-step-avx.S: New file.
	* gdb.arch/amd64-disp-step-avx.exp: New file.