Re: RFC: Should x86-64 support arbitrary calling conventions?

[Florian Weimer <fw at deneb dot enyo dot de>]

* H. J. Lu:

> According to x86-64 psABI, xmm0-xmm7 can be used to pass function
> parameters.  But ICC also uses xmm8-xmm15 to pass function
> parameters.  This doesn't work with newer glibc since
> _dl_runtime_resolve only preserves the first 8 vector registers:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=21236

Supporting truly arbitrary calling conventions pretty much requires
kernel support for a seamless context switch.

There's at least one compiler (MLton) which uses %rsp as a
general-purpose register because it does not use the hardware stack to
implement its function calls.  Surely we can't expect *that* to work
across shared object boundaries (and the compiler obviously does not,
although theoretically it could be make to work with two (indirect)
jumps instead of callq and retq).

There are always going to be some limitations on the calling
convention.  I would prefer if we could stick with what is in the
x86-64 psABI supplement, and not switch to some undocumented ad-hoc
ABI.  Because as I said, unless there is kernel support, there is
always going to be some form of ABI-imposed restriction on calling
conventions.