llvm.init.trampoline doesn't work if the function argument is a function pointer.

[llvmbot]

Bugzilla Link	8092
Version	trunk
OS	All
Attachments	A wrapper around llvm.init.trampoline
Reporter	LLVM Bugzilla Contributor

Extended Description

The attached .ll file contains a function calling llvm.init.trampoline, passing its own arguments on to it. LLVM doesn't like this:

$ llc make_trampoline.ll
llvm.init_trampoline parameter #​2 must resolve to a function.
%tramp = tail call i8* @​llvm.init.trampoline(i8* %mem, i8* %fn, i8* %data)
Broken module found, compilation aborted!
0 llc 0x0000000000da92df
1 llc 0x0000000000dab552
2 libpthread.so.0 0x00007fa4ab8b78f0
3 libc.so.6 0x00007fa4aaba6a75 gsignal + 53
4 libc.so.6 0x00007fa4aabaa5c0 abort + 384
5 llc 0x0000000000d2160b
6 llc 0x0000000000cfb6cd llvm::FPPassManager::runOnFunction(llvm::Function&) + 557
7 llc 0x0000000000cfb7cb llvm::FPPassManager::runOnModule(llvm::Module&) + 75
8 llc 0x0000000000cfb257 llvm::MPPassManager::runOnModule(llvm::Module&) + 503
9 llc 0x0000000000cfb3d7 llvm::PassManagerImpl::run(llvm::Module&) + 167
10 llc 0x000000000053744d main + 4109
11 libc.so.6 0x00007fa4aab91c4d __libc_start_main + 253
12 llc 0x0000000000535829
Stack dump:
0. Program arguments: llc make_trampoline.ll

1. Running pass 'Function Pass Manager' on module 'make_trampoline.ll'.
2. Running pass 'Module Verifier' on function '@make_trampoline'
   Aborted
   =====

Commenting out this assert in the verifier shows it is currently backed up by one in the SelectionDAGBuilder (by way of cast):

$ llc make_trampoline.ll
llc: /home/urxae/opt/llvm/trunk/include/llvm/Support/Casting.h:202: typename llvm::cast_retty<To, From>::ret_type llvm::cast(const Y&) [with X = llvm::Function, Y = llvm::Value*]: Assertion `isa(Val) && "cast() argument of incompatible type!"' failed.
0 llc 0x0000000000da928f
1 llc 0x0000000000dab502
2 libpthread.so.0 0x00007f771ea9d8f0
3 libc.so.6 0x00007f771dd8ca75 gsignal + 53
4 libc.so.6 0x00007f771dd905c0 abort + 384
5 libc.so.6 0x00007f771dd85941 __assert_fail + 241
6 llc 0x0000000000917abc llvm::SelectionDAGBuilder::visitIntrinsicCall(llvm::CallInst const&, unsigned int) + 17372
7 llc 0x0000000000917fae llvm::SelectionDAGBuilder::visitCall(llvm::CallInst const&) + 654
8 llc 0x00000000008ffb2a llvm::SelectionDAGBuilder::visit(unsigned int, llvm::User const&) + 394
9 llc 0x0000000000925d0d llvm::SelectionDAGBuilder::visit(llvm::Instruction const&) + 45
10 llc 0x0000000000933aa0 llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator<llvm::Instruction const>, llvm::ilist_iterator<llvm::Instruction const>, bool&) + 160
11 llc 0x00000000009340f4 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) + 1460
12 llc 0x0000000000934da6 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) + 486
13 llc 0x0000000000cfb6cd llvm::FPPassManager::runOnFunction(llvm::Function&) + 557
14 llc 0x0000000000cfb7cb llvm::FPPassManager::runOnModule(llvm::Module&) + 75
15 llc 0x0000000000cfb257 llvm::MPPassManager::runOnModule(llvm::Module&) + 503
16 llc 0x0000000000cfb3d7 llvm::PassManagerImpl::run(llvm::Module&) + 167
17 llc 0x000000000053744d main + 4109
18 libc.so.6 0x00007f771dd77c4d __libc_start_main + 253
19 llc 0x0000000000535829
Stack dump:
0. Program arguments: llc make_trampoline.ll

1. Running pass 'Function Pass Manager' on module 'make_trampoline.ll'.
2. Running pass 'X86 DAG->DAG Instruction Selection' on function '@make_trampoline'
   Aborted
   =====

Is there any reason for this?
As far as I can tell, there's no reason llvm.init.trampoline shouldn't be able to work for function pointers; the nest arg is given a specially-reserved register in the callling convention (at least on X86 and X86-64, I didn't check the other backends) so it's not like the argument list actually has to be rewritten by it. That means it should be able to just store some static code and the pointers to the memory provided, right?

(At the very least, this should be more clearly documented in LangRef.html if this was intentional; it does currently say "The func argument must hold a function bitcast to an i8*.", but that could be interpreted to mean "indirection" by way of function arguments, phi nodes, etc. is allowed)

[llvmbot]

The main problem here is... what is the calling convention? On x86, if using
the C calling convention then the 'nest' argument should be placed in ecx,
while with the X86_FastCall calling convention it goes in eax. It's not the
calling convention of the trampoline intrinsic itself that matters, it's the
calling convention of the underlying target function. This can be determined
automatically either by peeking at the called function (but in your case that
won't be possible) or looking at call sites which call the trampoline setup
by the intrinsic, which is not always possible. It's not clear to me what to
do here.

[llvmbot]

That's a good point. This would seem to be impossible without changing the IR representation of the trampoline intrinsic call sites.
So maybe the IR should change? If the calling convention was encoded in the intrinsic call it could be made to work, right?

Some possibilities:

1. Since the intrinsic isn't actually a actual function (or if it is, its calling convention is well-known), just set the calling convention of the target on the intrinsic (but eww, this is not a nice solution).

2. Add an extra parameter to llvm.init.trampoline(), so the call might be to e.g. @​llvm.init.trampoline(i8* %memptr, i8* %func, i8* %context, metadata !"fastcc"). Targets that don't care about the calling convention for this purpose (for instance x86_64, which always uses R10) can ignore this extra argument completely (except to assert() that it's a valid one for the target, hopefully).
   This requires CallingConv::ID <--> name mappings somewhere but the .ll parser and writer should already have those somewhere.

2.5) Similar to (2), but instead of using an MDString representation of the calling convention, an i32 encoding a CallingConv::ID could be passed.
This might be more convenient to users using the LLVM API but is probably much less clear to those reading or writing .ll files directly. (This requires those values to remain fixed for backward-compatibility, but this is already the case since they're encoded as integers in bitcode files when storing function prototypes, calls and invokes)

3. Encode the calling convention in the intrinsic name, e.g. @​llvm.init.trampoline.fastcc().
   This could even allow the old @​llvm.init.trampoline() to remain available with its current auto-deduction functionality, obviating the need to auto-upgrade.

(All of these would have undefined behavior if the calling convention specified doesn't match the actual calling convention and the resulting function pointer is called, of course)

Regarding backward compatibility:
Other than (1), these should be pretty easy to auto-upgrade (if the old intrinsic isn't kept with (3)) by moving the auto-deduction to the auto-upgrade logic (possibly after rewriting it to work on IR) instead of where it's done now (probably on SDNodes?).
(Option (1) might also be auto-upgradeable but would be trickier to pull off because the unupgraded call would still "look" like a valid upgraded call, except that the calling convention might not match the one of the function being passed in.)

Alternatively, you could do nothing except allowing function pointers to be passed in code for targets like X86_64 that don't actually care about the calling convention because they always use the same register. (But this might make it harder to auto-upgrade if a solution similar to one of the above-mentioned ones is ever implemented later)