Skip to content

[AArch64] Missed vectorisation opportunity (tsvc, s172) #71517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sjoerdmeijer opened this issue Nov 7, 2023 · 1 comment · Fixed by #80446
Closed

[AArch64] Missed vectorisation opportunity (tsvc, s172) #71517

sjoerdmeijer opened this issue Nov 7, 2023 · 1 comment · Fixed by #80446

Comments

@sjoerdmeijer
Copy link
Collaborator

We are not vectorising kernel s172 from TSVS and are 3x behind compared to GCC as a result. Compile this input with -O3 -ffast-math -mcpu=neoverse-v2:

__attribute__((aligned(64))) float x[32000];

__attribute__((aligned(64))) float a[32000],b[32000],c[32000],d[32000],e[32000],
                                   aa[256][256],bb[256][256],cc[256][256],tt[256][256];

int dummy(float[32000], float[32000], float[32000], float[32000], float[32000], float[256][256], float[256][256], float[256][256], float);

float  s172(int xa, int xb)
{
    int n1 = xa;
    int n3 = xb;

    for (int nl = 0; nl < 100000; nl++) {
        for (int i = n1-1; i < 32000; i += n3) {
            a[i] += b[i];
        }
        dummy(a, b, c, d, e, aa, bb, cc, 0.);
    }
}

Clang's codegen:

.LBB0_3:                                //   Parent Loop BB0_2 Depth=1
        ldr     s0, [x19, x8, lsl #2]
        ldr     s1, [x20, x8, lsl #2]
        fadd    s0, s1, s0
        str     s0, [x20, x8, lsl #2]
        add     x8, x8, x22
        cmp     x8, x23
        b.lt    .LBB0_3

GCC's codegen:

        whilelo p7.s, wzr, w28
.L5:
        ld1w    z31.s, p7/z, [x19, x0, lsl 2]
        ld1w    z30.s, p7/z, [x27, x0, lsl 2]
        fadd    z31.s, z31.s, z30.s
        st1w    z31.s, p7, [x19, x0, lsl 2]
        add     x0, x0, x2
        whilelo p7.s, w0, w28
        b.any   .L5

See also:
https://godbolt.org/z/W8eEPKqET

TODO: root cause analysis.

@llvmbot
Copy link
Member

llvmbot commented Nov 7, 2023

@llvm/issue-subscribers-backend-aarch64

Author: Sjoerd Meijer (sjoerdmeijer)

We are not vectorising kernel s172 from TSVS and are 3x behind compared to GCC as a result. Compile this input with `-O3 -ffast-math -mcpu=neoverse-v2`:
__attribute__((aligned(64))) float x[32000];

__attribute__((aligned(64))) float a[32000],b[32000],c[32000],d[32000],e[32000],
                                   aa[256][256],bb[256][256],cc[256][256],tt[256][256];

int dummy(float[32000], float[32000], float[32000], float[32000], float[32000], float[256][256], float[256][256], float[256][256], float);

float  s172(int xa, int xb)
{
    int n1 = xa;
    int n3 = xb;

    for (int nl = 0; nl &lt; 100000; nl++) {
        for (int i = n1-1; i &lt; 32000; i += n3) {
            a[i] += b[i];
        }
        dummy(a, b, c, d, e, aa, bb, cc, 0.);
    }
}

Clang's codegen:

.LBB0_3:                                //   Parent Loop BB0_2 Depth=1
        ldr     s0, [x19, x8, lsl #<!-- -->2]
        ldr     s1, [x20, x8, lsl #<!-- -->2]
        fadd    s0, s1, s0
        str     s0, [x20, x8, lsl #<!-- -->2]
        add     x8, x8, x22
        cmp     x8, x23
        b.lt    .LBB0_3

GCC's codegen:

        whilelo p7.s, wzr, w28
.L5:
        ld1w    z31.s, p7/z, [x19, x0, lsl 2]
        ld1w    z30.s, p7/z, [x27, x0, lsl 2]
        fadd    z31.s, z31.s, z30.s
        st1w    z31.s, p7, [x19, x0, lsl 2]
        add     x0, x0, x2
        whilelo p7.s, w0, w28
        b.any   .L5

See also:
https://godbolt.org/z/W8eEPKqET

TODO: root cause analysis.

ShivaChen added a commit to ShivaChen/llvm-project that referenced this issue Jan 8, 2024
… versioning

This commit enable the vectorization for the case from
llvm#71517.

  float  s172(int xa, int xb)  {
    for (int i = xa - 1; i < 32000; i += xb)
           a[i] += b[i];
  }

By assuming the stride as one and generating the runtime checking to guard
the vectorized loop, it seems the case can be vectorized.
fhahn added a commit that referenced this issue Jan 11, 2024
fhahn added a commit that referenced this issue Jan 23, 2024
End-to-end test for #71517,
testing IndVars/LoopVectorize interaction
justinfargnoli pushed a commit to justinfargnoli/llvm-project that referenced this issue Jan 28, 2024
fhahn added a commit to fhahn/llvm-project that referenced this issue Feb 2, 2024
We are replacing a narrow IV increment with a wider one. If the original
(narrow) increment did not wrap, the wider one should not wrap either. Set
the flags to be the union of both wide increment and original increment;
this ensures we preserve flags SCEV could infer for the wider increment.

Fixes llvm#71517.
fhahn added a commit that referenced this issue Feb 10, 2024
#80446)

We are replacing a narrow IV increment with a wider one. If the original
(narrow) increment did not wrap, the wider one should not wrap either.
Set the flags to be the union of both wide increment and original
increment; this ensures we preserve flags SCEV could infer for the wider
increment.

Fixes #71517.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants