Enabling LTO causes miscompilation on x86_64-apple-darwin

[briansmith]

This is from https://github.com/briansmith/ring 's CI, testing the tip of the main branch. The CI passes on 1.86 and fails on multiple targets when building with 1.87. In at least one case, building and linking passes but the test fails, indicating potentially the code (computing HMAC) or the test of the code is being miscompiled.

Unfortunately, I have limited access to systems that would help me reduce these today. I have noticed another project has run into a similar linker error recently. I apologize in advance for such a poor bug report. I hope I, or ideally others, could fill in the details, as I have limited availability today.

x86_64-apple-darwin

https://github.com/briansmith/ring/actions/runs/15143648178/job/42573984175

running 2 tests
test hmac_debug ... ok
test hmac_tests ... FAILED

failures:

---- hmac_tests stdout ----

thread 'hmac_tests' panicked at tests/hmac_tests.rs:83:9:
assertion `left == right` failed
  left: true
 right: false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

(The test is checking whether the output of the HMAC computation is correct.)

[briansmith]

See tikv/tikv#18465 for another project that seems to have encountered the same error with the same unrecognized relocation type error on ARM64 Linux, using nightly-2025-04-04.

[briansmith]

The problem seems to be caused by LTO, as when I remove "lto = true" from the release profile in Cargo.toml, things are working. See https://github.com/briansmith/ring/actions/runs/15145798156.

[lqd]

Can you bisect the PR causing this with cargo-bisect-rustc? Thanks!

[briansmith]

Can you bisect the PR causing this with cargo-bisect-rustc? Thanks!

If you are asking me, then no. It is very unlikely I will get to it, though in normal circumstances I would try.

[lqd]

The bisection should be helpful regardless of who's doing it, it doesn't have to be you, no worries.

[apiraino]

Assigning priority (discussion on Zulip).

@rustbot label -I-prioritize +P-critical +T-compiler

[briansmith]

My guess is that, for the x86_64-apple-darwin miscompile, there may be two issues: #140686, which we believe to be a linker bug (not sure if that is a bug in a very old linker, or a bug that also affects the linker that is used b by default in macos-13 GitHub runners, which are x86-64) and a codegen change that causes the linker bug to be hit more often. If so, reducing and bisecting might be more difficult than typical.

[dianqk]

A minimum reproduction: https://rust.godbolt.org/z/vd4nTc3oj

@rustbot label -E-needs-mcve

[justsmth]

As another data point, aws-lc-rs is seeing the same runtime failure on x86_64-apple-darwin on (essentially) that same test: https://github.com/aws/aws-lc-rs/actions/runs/15308838405/job/43076967009?pr=816#step:7:648

test hmac_debug ... ok
test hmac_traits ... ok
test hmac_tests ... FAILED
test hmac_thread_safeness ... ok

failures:

---- hmac_tests stdout ----

thread 'hmac_tests' panicked at aws-lc-rs/tests/hmac_test.rs:64:9:
assertion `left == right` failed
  left: true
 right: false

However, we're not seeing any failures during linking on the other platforms.

[workingjubilee]

I have split out the linkage issue to #141737

This issue only tracks the miscompilation now.

[workingjubilee]

We investigated this during the T-compiler meeting on 2025-05-29 and determined

• the failure here involves Xcode 15.2
  • it may not be linker-related
  • it may be a bug in ld-prime
  • it is probably not a bug in ld64, as we believe that linker was not used
• this is probably not the same bug as release builds using rustc 1.86.0 on macOS Ventura (intel) SDK exhibit incorrect behaviour #140686

[workingjubilee]

Let this be a reminder to everyone that every miscompilation issue is its own unique, sparkling star, and it is possible for two such stars, alike in dignity yet still standing in distinction, to fall from the heavenly tapestry of fate and strike the Rust toolchain at about the same time.

[dianqk]

Edit: All issues are temporarily resolved by llvm/llvm-project#142304, which will be included in the next LLVM update.

I have documented everything about #140686, #141306 and #141737 on https://github.com/dianqk/rust-140686-141306-141737. @apiraino We can mention it at the next meeting.

For #140686, I don't have any suggestion for cross-building for macOS on Linux. :
For #141306 (this one), P-critical probably can be removed after the next meeting, since Xcode 15.3 has fixed it.
For #141737, I have request reverting llvm/llvm-project#72584: llvm/llvm-project#72584 (comment).

[nikic]

@dianqk Are all issues here related to relative lookup tables?

[dianqk]

@dianqk Are all issues here related to relative lookup tables?

Yes, I didn't double-check #141306, but it's using the same minimal reproduction as #141737

[briansmith]

For #141306 (this one), P-critical can be removed after the next meeting, since upgrading to Xcode 15.3 is easy!

Is it easy? Many people are using macos-13 runners to ensure that cargo test runs their tests on macOS on native x86_64 CPUs. Last time I checked, macos-13 GitHub Runner images are the latest macOS x86_64 runners. And according to the documentation, 15.2 is the latest version installed on those runners. So it may not be "easy."

Also, Xcode 15.2 is the default Xcode version on those runners.

If we don't want to maintain compatibility with Xcode 15.2, that's fine. But, instead of people's code silently being miscompiled, we should at a minimum ensure that the build stops with an error when Xcode 15.2 (and earlier?) is used, or that LTO isn't used with its linker, or maybe something else. Anything other than silently miscompiling their code.

[dianqk]

Is it easy? Many people are using macos-13 runners to ensure that cargo test runs their tests on macOS on native x86_64 CPUs. Last time I checked, macos-13 GitHub Runner images are the latest macOS x86_64 runners. And according to the documentation, 15.2 is the latest version installed on those runners. So it may not be "easy."

Yes, you are right. Xcode 15.2 requires macOS 13.5+, but Xcode 15.3 requires macOS 14.0+.