This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: Ruby testsuite failures because of pointer mangling on 32-bit ARM?
- From: "Carlos O'Donell" <carlos at redhat dot com>
- To: Will Newton <will dot newton at linaro dot org>
- Cc: David Miller <davem at davemloft dot net>, Andrew Pinski <pinskia at gmail dot com>, libc-alpha <libc-alpha at sourceware dot org>
- Date: Fri, 13 Dec 2013 10:59:25 -0500
- Subject: Re: Ruby testsuite failures because of pointer mangling on 32-bit ARM?
- Authentication-results: sourceware.org; auth=none
- References: <20131210 dot 211901 dot 1840879367475720601 dot davem at davemloft dot net> <52A94FB5 dot 7060100 at redhat dot com> <20131212 dot 121441 dot 261870704107659875 dot davem at davemloft dot net> <20131212 dot 132053 dot 446942135510037848 dot davem at davemloft dot net> <52AA2BAB dot 50309 at redhat dot com> <CANu=DmgxLAsGkdnka1rYpE_aoTa79c18aua5D9tjFRYvA8vsvQ at mail dot gmail dot com>
On 12/12/2013 04:46 PM, Will Newton wrote:
> On 12 December 2013 21:33, Carlos O'Donell <carlos@redhat.com> wrote:
>> On 12/12/2013 01:20 PM, David Miller wrote:
>>> From: David Miller <davem@davemloft.net>
>>> Date: Thu, 12 Dec 2013 12:14:41 -0500 (EST)
>>>
>>>> From: "Carlos O'Donell" <carlos@redhat.com>
>>>> Date: Thu, 12 Dec 2013 00:55:01 -0500
>>>>
>>>>> It treats the jmp_buf as an array of VALUE sized pointers that
>>>>> it can examine to determine if there are pointers to the heap.
>>>>
>>>> Sounds similar to what any other garbage collector will do, scan
>>>> the processes address space looking for pointers.
>>>>
>>>> I'm pretty sure Boehm-GC does something similar, although perhaps
>>>> it scans the entire process stack from the point in which it is
>>>> called instead of using jmpbuf's to delineate spans of stack
>>>> areas like Ruby does.
>>>
>>> And, indirectly, realize that even a straight stack scan is going
>>> to potentially break if you start mangling pointers in jmpbuf.
>>>
>>> Consider the case where if the jmpbuf is on the processes stack, and
>>> normally it would get scanned by GC and the pointer followed to find
>>> memory references, and now that would not work because the pointer is
>>> mangled.
>>
>> Only if the jpmbuf pointer is the only way to access the other objects.
>> It seems like ruby is optimizing by starting with the jmpbuf to look
>> for object pointers.
>>
>>> I think all of these schemes are legitimate and erroneously broken
>>> by pointer mangling.
>>
>> The problem I have is that I can't objectively evaluate how well
>> pointer mangling is working as a security feature because I lack
>> any data about it. Therefore I can't compare it to the pain it would
>> cause ruby to reimplement GC using some other more costly scanning.
>>
>> What do we do here?
>
> If the frame pointer register is being used as a frame pointer it
> can't contain a data pointer. Likewise the stack pointer and lr. As
> far as i can tell the only issue is encrypting the frame pointer in
> the -fomit-frame-pointer case.
>
> Did the patch I submitted fix the ruby testsuite issue?
I'm still verifying this. Perhaps wait to checkin the patch until I give
an all clear?
Cheers,
Carlos.