CheckEnums tests fail on s390x (big-endian)

[fneddy]

Code

Some newly introduced tests from #141759 are failing on s390x (big endian)

rust/tests/ui/mir/enum/convert_non_enum_break.rs

Lines 1 to 20 in f51c987

	//@ run-fail
	//@ compile-flags: -C debug-assertions
	//@ error-pattern: trying to construct an enum from an invalid value 0x10000
	#[allow(dead_code)]
	#[repr(u32)]
	enum Foo {
	A,
	B,
	}
	#[allow(dead_code)]
	struct Bar {
	a: u16,
	b: u16,
	}
	fn main() {
	let _val: Foo = unsafe { std::mem::transmute::<_, Foo>(Bar { a: 0, b: 1 }) };
	}

rust/tests/ui/mir/enum/niche_option_tuple_break.rs

Lines 1 to 20 in f51c987

	//@ run-fail
	//@ compile-flags: -C debug-assertions
	//@ error-pattern: trying to construct an enum from an invalid value 0x3
	#[allow(dead_code)]
	enum Foo {
	A,
	B,
	}
	#[allow(dead_code)]
	struct Bar {
	a: usize,
	b: usize,
	}
	fn main() {
	let _val: Option<(usize, Foo)> =
	unsafe { std::mem::transmute::<_, Option<(usize, Foo)>>(Bar { a: 3, b: 3 }) };
	}

rust/tests/ui/mir/enum/convert_non_enum_ok.rs

Lines 1 to 20 in f51c987

	//@ run-pass
	//@ compile-flags: -C debug-assertions
	#[allow(dead_code)]
	#[repr(u32)]
	enum Foo {
	A,
	B,
	}
	#[allow(dead_code)]
	struct Bar {
	a: u16,
	b: u16,
	}
	fn main() {
	let _val: Foo = unsafe { std::mem::transmute::<_, Foo>(Bar { a: 0, b: 0 }) };
	let _val: Foo = unsafe { std::mem::transmute::<_, Foo>(Bar { a: 1, b: 0 }) };
	}

rust/tests/ui/mir/enum/with_niche_int_break.rs

Lines 1 to 21 in f51c987

	//@ run-fail
	//@ compile-flags: -C debug-assertions
	//@ error-pattern: trying to construct an enum from an invalid value 0x4
	#[allow(dead_code)]
	#[repr(u16)]
	enum Mix {
	A,
	B(u16),
	}
	#[allow(dead_code)]
	enum Nested {
	C(Mix),
	D,
	E,
	}
	fn main() {
	let _val: Nested = unsafe { std::mem::transmute::<u32, Nested>(4) };
	}

I expected to see this happen:

the tests should pass.

Instead, this happened:

the tests fail.

Version it worked on

It most recently worked on: never

Version with regression

./build/s390x-unknown-linux-gnu/stage1/bin/rustc --version
rustc 1.90.0-nightly (6988a8fea 2025-07-01)
./build/s390x-unknown-linux-gnu/stage2/bin/rustc --version
rustc 1.90.0-nightly (6988a8fea 2025-07-01)

Backtrace

none

Analysis

it looks like this are all the tests where transmute is transmuting between different sized types. E.g. the enum is sized different as the container data that is used to fill it. i looked at the MIR code and it looks like it is producing the same code for for x86_64(little endian) and s390x(big endian)

E.g. niche_option_tuple_break.rs:

cat niche_option_tuple_break.mir
// WARNING: This output format is intended for human consumers only
// and is subject to change without notice. Knock yourself out.
// HINT: See also -Z dump-mir for MIR at specific points during compilation.
fn main() -> () {
    let mut _0: ();
    let _1: std::option::Option<(usize, Foo)>;
    let mut _2: Bar;
    let mut _3: [std::mem::MaybeUninit<u8>; 16];
    let mut _4: u8;
    let mut _5: u128;
    let mut _6: u128;
    let mut _7: bool;
    scope 1 {
        debug _val => _1;
    }

    bb0: {
        _2 = Bar { a: const 3_usize, b: const 3_usize };
        _3 = copy _2 as [std::mem::MaybeUninit<u8>; 16] (Transmute);
        _4 = copy _3[8..9] as u8 (Transmute);
        _5 = copy _4 as u128 (IntToInt);
        _6 = Sub(copy _5, const 0_u128);
        _7 = Le(copy _6, const 2_u128);
        assert(copy _7, "trying to construct an enum from an invalid value {}", copy _5) -> [success: bb1, unwind unreachable];
    }

    bb1: {
        _1 = move _2 as std::option::Option<(usize, Foo)> (Transmute);
        return;
    }
}

I guess here is the problem: _4 = copy _3[8..9] as u8 (Transmute);

On big-endian the copy will just a take a zero byte from _3 to copy into _4. Therefore the assert will not hit and as a result the test fails.

To cross-verify, if the problem is only in the debug-assertion, or if casting like this just plainly does not work, i changed the test slightly:

#[allow(dead_code)]
#[derive(Debug,Clone)]
enum Foo {
    A,
    B,
}

#[allow(dead_code)]
#[derive(Debug,Clone)]
struct Bar {
    a: usize,
    b: usize,
}

fn main() {
    let mut _val: Option<(usize, Foo)> =
        unsafe { std::mem::transmute::<_, Option<(usize, Foo)>>(Bar { a: 3, b: 1 }) };

    println!("{:#?}",_val.clone().unwrap());
    println!("{:#?}",_val.clone().unwrap().1 as usize);
}

this prints on x86:

(
    3,
    B,
)
1

but on s390x:

(
    3,
    A,
)
0

Result

It seems that transmuting like this on big-endian is not supported?

Either this is expected. But in that case the complete debug-assertion does not make sense on big-endian system.

Or the transmute should have worked correctly and figured out the byte ordering. But in that case I guess that this never really worked.
-->

[rustbot]

Unknown labels: regression-from-stable-to-{channel}

[saethlin]

Just FYI, a good way to check if this is a bug in the new checks is to see if it disagrees with Miri.

[compiler-errors]

cc @1c3t3a

[1c3t3a]

Thanks for bringing this to my attention! And thanks for the excellent bug report!

Yeah I was not paying enough attention when writing the test suite it seems. As @saethlin mentioned, I executed this in Miri and it has the same output as what the enum check gives (which is different then the test expecations which leads to the failure).

[saethlin]

So this isn't a regression, just a testsuite issue.