[commit/powerpc] crash trying to allocate memory in inferior

[Joel Brobecker <brobecker at adacore dot com>]

Our testsuite noticed a crash when trying to call a function which
requires GDB to allocate memory in the inferior. Typically, this
happens when one of the parameters is a string.  For instance, our
testcase tries:

    (gdb) call debug.trace (me, "You")
    [1]    32737 segmentation fault /path/to/gdb

What happens is that GDB sees the string, and thus tries to allocate
memory for it in the inferior:

> /* Allocate NBYTES of space in the inferior using the inferior's
>    malloc and return a value that is a pointer to the allocated
>    space.  */
>
> struct value *
> value_allocate_space_in_inferior (int len)
> {
>   struct objfile *objf;
>   struct value *val = find_function_in_inferior ("malloc", &objf);

And find_function_in_inferior first searches the symtab in case
we have debug info.  But, in our case (bareboard powerpc), we don't,
so it gets "malloc"'s address from the minimal symbols, and builds
a value whose type is a TYPE_CODE_PTR, not a TYPE_CODE_FUNC.

As a result, when we later try to make the call to malloc, we end up
inside the powerpc tdep code that has:

> do_ppc_sysv_return_value (struct gdbarch *gdbarch, struct type *func_type,
[...]
>   if (func_type
>       && TYPE_CALLING_CONVENTION (func_type) == DW_CC_GDB_IBM_OpenCL)

The problem is that func_type is not a TYPE_CODE_FUNC, and thus
the type-specific kind is not TYPE_SPECIFIC_FUNC, and so we do
TYPE_CALLING_CONVENTION is an invalid access.

Interestingly, the other call to TYPE_CALLING_CONVENTION is correctly
preceded by a check of the type's TYPE_CODE (making sure that it is
TYPE_CODE_FUNC).

gdb/ChangeLog:

        * ppc-sysv-tdep.c (do_ppc_sysv_return_value): Do not check
        FUNC_TYPE's calling convention if FUNC_TYPE is not a function.

tested on powerpc-elf. Checked in.

---
 gdb/ChangeLog       |    5 +++++
 gdb/ppc-sysv-tdep.c |    1 +
 2 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 30cf144..f5cdd45 100644
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,3 +1,8 @@
+2011-10-24  Joel Brobecker  <brobecker@adacore.com>
+
+	* ppc-sysv-tdep.c (do_ppc_sysv_return_value): Do not check
+	FUNC_TYPE's calling convention if FUNC_TYPE is not a function.
+
 2011-10-24  Pedro Alves  <pedro@codesourcery.com>
 
 	* linux-nat.c (linux_handle_extended_wait): When handling a clone
diff --git a/gdb/ppc-sysv-tdep.c b/gdb/ppc-sysv-tdep.c
index e431363..bda4544 100644
--- a/gdb/ppc-sysv-tdep.c
+++ b/gdb/ppc-sysv-tdep.c
@@ -692,6 +692,7 @@ do_ppc_sysv_return_value (struct gdbarch *gdbarch, struct type *func_type,
   int opencl_abi = 0;
 
   if (func_type
+      && TYPE_CODE (func_type) == TYPE_CODE_FUNC
       && TYPE_CALLING_CONVENTION (func_type) == DW_CC_GDB_IBM_OpenCL)
     opencl_abi = 1;
 
-- 
1.7.1