Re: Large-Address awareness on 64 bit systems

[Ryan Johnson <ryan dot johnson at cs dot utoronto dot ca>]

On 18/06/2011 4:17 PM, Corinna Vinschen wrote:
> I just read something interesting on MSDN which only affects 64 bit systems:
>
>   "If the application has the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the
>    image header, each 32-bit application receives 4 GB of virtual address
>    space in the WOW64 environment.  If the IMAGE_FILE_LARGE_ADDRESS_AWARE
>    flag is not set, each 32-bit application receives 2 GB of virtual address
>    space in the WOW64 environment."
>
> So, with this flag set, a WOW64 process has the full 4 Gigs address
> space and none of that is taken by the OS?  That sounds cool.
I believe Solaris does this as well when running 32-bit apps. Kernel 
memory is all mapped past 4GB and thus invisible until a trap/syscall 
puts the CPU back into 64-bit mode.

> That means, if we make all Cygwin binaries large address aware, somebody
> using a 64 bit system could rebase all Cygwin DLLs, except for the
> Cygwin DLL itself into the virtual memory area beyond 0x80000000.  Given
> that this memory area is not at all used by Windows itself, it's free
> for usage by Cygwin alone.  This would affect the memory allocation not
> only for Cygwin DLLs, but also for mmaps and for thread stacks, which
> both use the MEM_TOP_DOWN flag.
Wouldn't we want thread stacks to stay in the low 2GB, given Windows' 
bad habit of moving them around? In general, how does ASLR affect 
things? Do heaps and system dlls risk wandering into high memory?

> it might be quite tricky to maintain a sane system, given that
> gcc creates executables with the large address awareness flag switched
> off, and given the fact that the flag is reset for a distro executable
> every time it gets updated via setup.
It seems like a small shell script could be a drop-in replacement for 
gcc, calling gcc and peflags under the hood. However, rebase[all] is a 
bit trickier. I would imagine that gcc's auto basing feature would not 
target high memory addresses.

> However, if anybody feels confident to test this on the own machine,
> I'd be curious if that has the desired effect of letting fork work
> more reliable.  I'm not sure I have the required use cases to test
> this sufficiently.
Test cases I can think of that really expose the weaknesses of fork:
- perl, python, and friends which use tons of dll modules
- lisp-like languages which dynamically compile and load code
- bootstrapping a gcc compiler
- deliberately dlopen-ing two dlls with the same base address and trying 
to fork

The first case tests fork's robustness for known dlls; the middle two 
uncover issues with dlls that are created on the fly; the last is just 
mean and nasty, in spite of its simplicity.

Ryan