[Bug gdb/19250] New: When enabling build-with-cxx, ptrace args type are not detected properly

["simon.marchi at ericsson dot com" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=19250

            Bug ID: 19250
           Summary: When enabling build-with-cxx, ptrace args type are not
                    detected properly
           Product: gdb
           Version: HEAD
            Status: NEW
          Severity: normal
          Priority: P2
         Component: gdb
          Assignee: unassigned at sourceware dot org
          Reporter: simon.marchi at ericsson dot com
  Target Milestone: ---

The strategy used right now to detect ptrace argument types is to try to
re-declare the ptrace function with various argument combinations until it
works. This relies on the fact that a function can't be re-declared with
different arguments in C.

For example, ptrace is declared like this in OpenBSD:

  int ptrace(int _request, pid_t _pid, caddr_t _addr, int _data);

When the configure test tries to re-declare it as, let's say,

  int ptrace(long, int, int, int);

the compilation fails and it knows that this isn't the right combination of
arguments.  When building with --enable-build-with-cxx, that test is done with
the c++ compiler (see 54019719152ab269fb4cec2c6a8a245ba6af6e49).  This strategy
simply doesn't work in C++, since it's valid to re-declare a function with
different parameters (because overloading is allowed).  So the first arguments
combination is always considered the right one and is used.  The build fails
later when ptrace is actually used.


Here are some possible ideas for a solution.

- We only need to use C++ for the first part of the test, to properly detect if
the first argument is an enum or not.  We could revert to C for the second part
of the test.

- Instead of just re-declaring the function, try to compile a call to it with
casts to the types to try:

  ptrace((long) 0, (int) 0, (int) 0, (int) 0)

  However, it might not work because of implicit silent conversion between int
types.

- Use extern "C" to re-declare ptrace. It wasn't obvious to try, because the
AC_TRY_COMPILE autoconf macro inserts the code in the body of main, and it
doesn't seem valid to use extern "C" in that scope.  The declaration should be
moved outside, to the global scope. That would require using another (probably
lower level) autoconf macro.

-- 
You are receiving this mail because:
You are on the CC list for the bug.