AVX mem broadcasts are cached on the stack

[KyleSiefring]

After exhausting registers inside of a loop, clang stores the results of a broadcast on the stack. This is inefficient, since broadcasting from memory is as fast as loading

Consider the following pseudo code:

float *restrict arr = ...; // prevent aliasing
loop {
     exhaust vector registers
     __mm256 x = _mm256_set1_ps(arr[0]);
     use x
}

When clang compiles this, arr[0] is broadcasted outside the loop then x is stored on the stack.

        vbroadcastss    ymm0, dword ptr [rdx]
        vmovups ymmword ptr [rsp - 72], ymm0
loop:
        ...
        load x from stack
        use x
        jmp loop

The expected behavior is:

loop:
       ...
        vbroadcastss    x, dword ptr [rdx]
        use x
        jmp loop

Obligatory Godbolt Sample: https://godbolt.org/z/v7MYcefxY (Sorry if my method of stressing register allocation results in too much asm/bytecode.)

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: Kyle Siefring (KyleSiefring)

After exhausting registers inside of a loop, clang stores the results of a broadcast on the stack. This is inefficient, since broadcasting from memory is as fast as loading

Consider the following pseudo code:

float *restrict arr = ...; // prevent aliasing
loop {
     exhaust vector registers
     __mm256 x = _mm256_set1_ps(arr[0]);
     use x
}

When clang compiles this, arr[0] is broadcasted outside the loop then x is stored on the stack.

        vbroadcastss    ymm0, dword ptr [rdx]
        vmovups ymmword ptr [rsp - 72], ymm0
loop:
        ...
        load x from stack
        use x
        jmp loop

The expected behavior is:

loop:
       ...
        vbroadcastss    x, dword ptr [rdx]
        use x
        jmp loop

Obligatory Godbolt Sample: https://godbolt.org/z/v7MYcefxY (Sorry if my method of stressing register allocation results in too much asm/bytecode.)

[RKSimon]

This has been on the backlog for a long time now - for constants at least we made progress by adding X86FixupVectorConstantsPass, and I started work on removing constant pool broadcasts from DAG entirely with #73509 - but addressing all the regressions for AVX512VL is a slog and handling the regressions for basic AVX was even worse (plus we need to handle optsize constants cases).

For non-constant loads, it might be that we can add a tweak to MachineLICM for loop hoisting cases as a starting point.

[KyleSiefring]

After some further poking at how zen cpus work, I'm not certain this is as bad an approach as I thought. Broadcasts from memory are micro-oped on zen, so this approach may even be preferable on that platform for anything but 512-bit wide vectors.

I still think it's less than ideal, since it is pretty bad on intel's e-cores.