Re: [RFC] nptl: change default stack guard size of threads

[James Greenhalgh <james dot greenhalgh at arm dot com>]

On Tue, Dec 05, 2017 at 10:55:31AM +0000, James Greenhalgh wrote:
> Thanks for your advice so far. To reiterate, I'm not pushing any particular
> optimization agenda in GCC, but I would like to understand the trade-off
> we're discussing.

Thank you for all the responses in this thread. I've certainly got a much
clearer view of the competing security, performance and correctness concerns
having read through the comments here.

One thing is very clear, we all want to ensure that the stack clash
protection is applied in as many cases as possible to ensure integrity of
the whole system security.

I see our trade-offs on LP64 AArch64 like this:

Option 1: 64k guard pages for LP64 on AArch64.

  * Better for performance
  * Almost all functions (99.981%) can handle their initial stack setup
      with a single probe
    ** 0.019% require multiple probes.
  * Performance impact unlikely to discourage use - GCC can enable by default.
  * Requires either 64k pages on the system, or for glibc to be rewritten
      to accommodate guard pages which are a multiple of the physical page size.
    ** Note: For small (Raspberry Pi sized) systems, 64k pages are considered
          inappropriate by distributions like fedora.
  * Likely to require similar rewrites for other C libraries, which also
      assume physical page-size guard pages.
  * Cannot give 100% guarantees about safety when considering
      backwards-compatibility and custom user-allocated stacks.
  * Divergent from ILP32 where we will likely want 4k.

Option 2: 4k guard pages for LP64 for AArch64

  * Worse for performance
  * 99.647% of functions can handle initial stack setup with a single probe
    ** 0.353% would require multiple probes
  * Performance impact may discourage users from enabling by default
  * No requirement on system (guard page size == minimal physical page size)
  * No requirement to rewrite C functions.
  * No divergence for ILP32

The fundamental disagreement is on what leads to a higher chance of achieving
complete protection - if the performance overhead is high, will our users
choose to disable the protection, if we keep the performance overhead low
do we lose protections for corner-cases like user-allocated stacks and older
C libraries.

And honestly, I don't see an easy way to draw a line here. Both sides have
clear merit and cover a slightly different use case. This should be a
distribution-level choice not a GCC policy.

Our proposal then, having spoken things through with the Arm engineers
here, and taken in to consideration the opinions on this thread, is that
we move to two "blessed" configurations of the GCC support for AArch64.

One would assume 64k guard pages. This would work out-of-the-box on systems
with a 64k page size, and would work with modifications to glibc (and other
libraries as appropriate) on systems with a 4k (or other) page size. The
performance impact will be low. If we take this approach, this will be the
default configuration for GCC.

The other would assume 4k guard pages. This would work everywhere, and
as-good-as guarantee complete coverage. However, it would be inefficient
on systems with a larger page size, or with a glibc upgraded to use
64k guard-pages.

My opinion is that leaving this configurable, with a sensible and low
overhead default is the best way to make forward progress in the diverse
AArch64 ecosystem. A distribution can then always choose to configure GCC to
assume 4k should they wish to. Naturally, this split model has advantages
for our ILP32 support, which could now reasonably default to the 4k mode.

That will still require some work on the glibc side to make the 64k guard
page on a 4k page system work, and will require extra work on the GCC side
(not required for Jeff's patches to be merged, but highly desirable for the
GCC 8 release) to set up the compiler configuration defaults.

I owe Jeff more review on the GCC side, but I hope this addresses one of the
key design differences and allows us to move forwards.

Please do let me know what I've missed and step in or if I'm still pushing
for a Very Bad Thing :-).

Thanks,
James