add `extern "custom"` functions #140770

folkertdev · 2025-05-07T20:26:31Z

tracking issue: #140829
previous discussion: #140566

In short, an extern "custom" function is a function with a custom ABI, that rust does not know about. Therefore, such functions can only be defined with #[unsafe(naked)] and naked_asm!, or via an extern "C" { /* ... */ } block. These functions cannot be called using normal rust syntax: calling them can only be done from inline assembly.

The motivation is low-level scenarios where a custom calling convention is used. Currently, we often pick extern "C", but that is a lie because the function does not actually respect the C calling convention.

At the moment "custom" seems to be the name with the most support. That name is not final, but we need to pick something to actually implement this.

r? @traviscross
cc @tgross35

rustbot · 2025-05-07T20:26:39Z

These commits modify compiler targets.
(See the Target Tier Policy.)

rustbot · 2025-05-07T20:53:08Z

Some changes occurred in compiler/rustc_codegen_cranelift

cc @bjorn3

Some changes occurred in compiler/rustc_codegen_gcc

cc @antoyo, @GuillaumeGomez

compiler/rustc_smir/src/stable_mir/abi.rs

compiler/rustc_passes/src/custom_abi.rs

compiler/rustc_passes/messages.ftl

compiler/rustc_smir/src/stable_mir/abi.rs

traviscross · 2025-05-08T02:49:24Z

tests/ui/abi/custom.rs

+#[unsafe(naked)]
+unsafe extern "custom" fn double(a: u64) -> u64 {
+    naked_asm!("add rax, rax", "ret");
+}


For now, I think I'd suggest just not allowing parameters and a return type -- or putting this under a separate feature gate.

We could always allow this later. It seems like one of the few question marks here, so it'd be better to do the minimal thing to start.

tests/ui/abi/bad-custom.rs

bjorn3 · 2025-05-08T18:39:10Z

Emit some extern "custom" errors in ast_validation:wq

Did you forget to press escape before quitting vim?

folkertdev · 2025-05-08T18:45:19Z

ah, yeah. Well it'll probably get squashed before this is ready

folkertdev · 2025-05-08T20:59:28Z

I've made a proper tracking issue at #140829

bors · 2025-05-17T07:02:22Z

☔ The latest upstream changes (presumably #141002) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-06-04T03:24:22Z

☔ The latest upstream changes (presumably #141984) made this pull request unmergeable. Please resolve the merge conflicts.

folkertdev · 2025-06-05T21:36:54Z

I've rebased and changed the implementation to match #141774, which is both shorter and apparently faster too.

bors · 2025-06-06T16:05:55Z

☔ The latest upstream changes (presumably #142099) made this pull request unmergeable. Please resolve the merge conflicts.

folkertdev · 2025-06-06T16:56:51Z

@traviscross are we waiting for T-lang for something here? Otherwise we could get a T-compiler review to move this forward right?

bors · 2025-06-06T22:07:31Z

☔ The latest upstream changes (presumably #141774) made this pull request unmergeable. Please resolve the merge conflicts.

workingjubilee · 2025-06-07T03:27:01Z

I believe T-lang has already basically okayed this experiment per TC's comment here and I think any concerns that TC winds up having, in addition to the review TC already gave, can be addressed in followup. It's not like lang's gonna FCP this tomorrow, is it? Rolling a compiler review.

r? compiler

workingjubilee

I'm sorry but I have a huge and tedious nit that should be a tidy rule but isn't because Reasons.

compiler/rustc_abi/src/canon_abi.rs

compiler/rustc_ast_passes/src/errors.rs

compiler/rustc_codegen_cranelift/src/abi/mod.rs

compiler/rustc_codegen_gcc/src/abi.rs

compiler/rustc_codegen_llvm/src/abi.rs

compiler/rustc_target/src/spec/abi_map.rs

traviscross · 2025-06-07T05:15:47Z

Yes, this is OK as a lang matter. I'd probably suggest...

r? tgross35

tgross35 · 2025-06-08T02:40:54Z

Happy to review on Monday, but @workingjubilee if you've already taken a look at the rest then feel free to pick it up as well

bors · 2025-06-08T02:49:25Z

☔ The latest upstream changes (presumably #142181) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2025-06-09T09:11:48Z

☔ The latest upstream changes (presumably #141435) made this pull request unmergeable. Please resolve the merge conflicts.

tgross35

This looks pretty clean to me, and well tested. Left a handful of small comments but nothing major.

compiler/rustc_ast/src/ast.rs

tests/ui/abi/bad-custom.rs

tgross35 · 2025-06-10T03:02:34Z

compiler/rustc_codegen_cranelift/src/abi/mod.rs

+        // Functions with this calling convention can only be called from assembly, but it is
+        // possible to declare an `extern "custom"` block, so the backend still needs a calling
+        // convention for declaring foreign functions.
+        CanonAbi::Custom => default_call_conv,


This seems reasonable but I wonder if down the line clif (or the other backends) might want to have some kind of calling convention representation that can be bug!ed on if it gets directly called somewhere. (cc @bjorn3)

tests/ui/abi/custom.rs

compiler/rustc_hir_typeck/src/callee.rs

tgross35 · 2025-06-10T03:23:59Z

compiler/rustc_hir_typeck/src/callee.rs

+        let abi = match callee_ty.kind() {
+            ty::FnDef(def_id, _) => self.tcx.fn_sig(def_id).skip_binder().skip_binder().abi,
+            ty::FnPtr(_, header) => header.abi,
+            _ => return,


Are there other valid callees? If so, it may be good to return from them specifically then bug! on the fallback _.

Hmm, that raises the question of what async extern "custom" fn foo() {} should do. It always returns impl Future, so I think we should just disallow this.

Agreed to disallowing async 👍 that's likely pretty tied to the Rust ABI too.

@folkertdev could you make the fallback bug so this doesn't miss any future call types?

ty::FnDef(def_id, _) => self.tcx.fn_sig(def_id).skip_binder().skip_binder().abi, ty::FnPtr(_, header) => header.abi, // Closures cannot have the "custom" ABI ty::Closure(..) => return, // Ignore errors ty::Error(_) => return, _ => span_bug!(span, "unexpected callee type {callee_ty:?}"),

(skipping errors based on

rust/compiler/rustc_type_ir/src/ty_kind.rs

Line 276 in 8ce2287

pub fn fn_sig(self, interner: I) -> ty::Binder<I, ty::FnSig<I>> {

)

Ah, I did look into that but it turns out that we do get (at least) Adt and Alias in that final branch. So that'll need some investigation.

I believe that what can happen here is that the scrutinee is some impl Fn() (or similar) value: it is callable, but not a function definition or function pointer.

@oli-obk you've been working in this area a bunch: do you have advice on getting the ABI here (anything that implicitly uses the extern "Rust" (closures, Fn()) is irrelevant, we're just looking for callable items with the "custom" ABI. I suspect &unsafe extern "custom" fn() in particular is going to be tricky and is missed by the current logic. Is there some standard way to get at the "real" type?

compiler/rustc_hir_analysis/messages.ftl

tgross35 · 2025-06-10T23:27:34Z

tests/ui/abi/bad-custom.rs

Based on #140770 (comment) could you add a test for an async function and a const function? Const is rejected for calling asm (will probably always be that way) and async seems to be rejected by checks on the form it desugars to, but it would be good to make sure these don't regress.

https://play.rust-lang.org/?version=nightly&mode=debug&edition=2024&gist=2823536b9788410319e9f67b66a480a1

bors · 2025-06-11T03:40:42Z

☔ The latest upstream changes (presumably #141883) made this pull request unmergeable. Please resolve the merge conflicts.

rustbot assigned traviscross May 7, 2025

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 7, 2025

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from f61cada to 847b357 Compare May 7, 2025 20:53

bjorn3 reviewed May 7, 2025

View reviewed changes

compiler/rustc_smir/src/stable_mir/abi.rs Show resolved Hide resolved

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from 847b357 to 93971b9 Compare May 7, 2025 22:17

tgross35 reviewed May 7, 2025

View reviewed changes

compiler/rustc_passes/src/custom_abi.rs Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

mejrs reviewed May 7, 2025

View reviewed changes

compiler/rustc_passes/src/custom_abi.rs Outdated Show resolved Hide resolved

compiler/rustc_passes/messages.ftl Outdated Show resolved Hide resolved

compiler/rustc_smir/src/stable_mir/abi.rs Show resolved Hide resolved

traviscross reviewed May 8, 2025

View reviewed changes

tgross35 reviewed May 8, 2025

View reviewed changes

tests/ui/abi/bad-custom.rs Outdated Show resolved Hide resolved

folkertdev force-pushed the custom-abi branch from 93971b9 to a72606d Compare May 8, 2025 15:44

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from a72606d to 64868d3 Compare May 8, 2025 17:02

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from 64868d3 to 7807720 Compare May 8, 2025 17:59

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from 7807720 to 9d529f8 Compare May 8, 2025 18:37

folkertdev mentioned this pull request May 8, 2025

Tracking Issue for abi_custom #140829

Open

3 tasks

folkertdev force-pushed the custom-abi branch from 9d529f8 to 6ebe16c Compare May 8, 2025 20:58

folkertdev force-pushed the custom-abi branch from 6ebe16c to f9e1963 Compare May 17, 2025 19:22

folkertdev force-pushed the custom-abi branch from 4c967f4 to a705a09 Compare June 5, 2025 21:31

This comment has been minimized.

Sign in to view

folkertdev force-pushed the custom-abi branch from a705a09 to 5c5e04e Compare June 5, 2025 21:44

folkertdev force-pushed the custom-abi branch from 5c5e04e to 6073b4c Compare June 6, 2025 16:39

folkertdev force-pushed the custom-abi branch from 6073b4c to e433f07 Compare June 6, 2025 23:09

rustbot assigned fee1-dead and unassigned traviscross Jun 7, 2025

workingjubilee added A-naked Area: `#[naked]`, prologue and epilogue-free, functions, https://git.io/vAzzS A-ABI Area: Concerning the application binary interface (ABI) labels Jun 7, 2025

workingjubilee requested changes Jun 7, 2025

View reviewed changes

This comment was marked as resolved.

Sign in to view

rustbot assigned tgross35 and unassigned fee1-dead Jun 7, 2025

folkertdev force-pushed the custom-abi branch from e433f07 to 675d6e6 Compare June 7, 2025 10:40

folkertdev force-pushed the custom-abi branch from 675d6e6 to a3fe4a3 Compare June 8, 2025 09:26

folkertdev force-pushed the custom-abi branch from a3fe4a3 to 823e466 Compare June 9, 2025 10:23

tgross35 reviewed Jun 10, 2025

View reviewed changes

add extern "custom" functions

3052371

folkertdev force-pushed the custom-abi branch from 823e466 to 3052371 Compare June 10, 2025 09:46

tgross35 reviewed Jun 10, 2025

View reviewed changes

add extern "custom" functions #140770

Are you sure you want to change the base?

add extern "custom" functions #140770

Conversation

folkertdev commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented May 7, 2025

Uh oh!

This comment has been minimized.

rustbot commented May 7, 2025

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bjorn3 commented May 8, 2025

Uh oh!

folkertdev commented May 8, 2025

Uh oh!

folkertdev commented May 8, 2025

Uh oh!

bors commented May 17, 2025

Uh oh!

bors commented Jun 4, 2025

Uh oh!

folkertdev commented Jun 5, 2025

Uh oh!

This comment has been minimized.

bors commented Jun 6, 2025

Uh oh!

folkertdev commented Jun 6, 2025

Uh oh!

bors commented Jun 6, 2025

Uh oh!

workingjubilee commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workingjubilee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

traviscross commented Jun 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

tgross35 commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bors commented Jun 8, 2025

Uh oh!

bors commented Jun 9, 2025

Uh oh!

tgross35 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

add `extern "custom"` functions #140770

add `extern "custom"` functions #140770

folkertdev commented May 7, 2025 •

edited

Loading

workingjubilee commented Jun 7, 2025 •

edited

Loading

traviscross commented Jun 7, 2025 •

edited

Loading

tgross35 commented Jun 8, 2025 •

edited

Loading