Introduce CallConvMemberFunction to represent the member-function variants of Cdecl, Stdcall, etc. on Windows #46775
Description
Background and Motivation
For interop developers of managed APIs wrapping native API layers, sometimes they need to wrap C++ instance member functions with non-default calling conventions like cdecl and stdcall. However, on Windows, the C++ instance member function calling convention for cdecl and stdcall is not the same as the non-member function calling convention. I propose that we add a new calling convention type that can be used in the extensible calling convention syntax to denote that the function pointer should use the instance member calling convention variant of the specified calling convention instead of the non-member function calling convention.
Proposed API
namespace System.Runtime.CompilerServices
{
+ public class CallConvMemberFunction {
+ }
}Usage Examples
This API would be used in the function pointer syntax as follows:
S Foo(void* thisPtr, int i, int j, delegate* unmanaged[Stdcall, MemberFunction]<void*, int, int, S> f)
{
return f(thisPtr, i, j);
}The MemberFunction modifier would modify the provided calling convention, or modify the default calling convention if MemberFunction or another calling convention modifier like SuppressGCTransition were provided.
If the MemberFunction is used on a platform where there is no "member function calling convention variant" of the other provided calling conventions, then it will be ignored.
Alternative Designs
We considered instead of introducing a new API, allowing function pointers of the style delegate* unmanaged[Stdcall, Thiscall]<void*, int, int, S> instead of introducing a new type. The CoreCLR Interop team decided that we did not like this design because it is not straightforward which type is the calling convention and which is a modifier. Additionally, allowing the above form would also require us to allow delegate* unmanaged[Thiscall, Stdcall]<void*, int, int, S> since they are equivalent, which we'd highly prefer not to do. We prefer a model where a function pointer can have at most one of the CallConv*call types provided to specify the "base" calling convention and any number of other CallConv* types that don't end in "call", such as this one or CallConvSuppressGCTransition, that modify the "base" calling convention.
We also considered alternative names like CallConvInstanceMethod or CallConvInstanceMemberFunction.
After more discussion, we've also come up with another alternative name that doesn't use C++ terminology so as to avoid implying more fully-featured C++ interop support: After even further discussion, this was also determined undesirable due to lack of clarity and since the weirdness that calling conventions account for is inherently something specific to (Windows) C++.CallConvStructReturnBuffer. This name specifies the exact modification in the calling convention (structs go into return buffers) that happens on Windows in the cases this API proposal is meant to cover.