Bump PyTorch nightly pin past April 22, 2025

[swolchok]

Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .

[swolchok]

Stack from ghstack (oldest at bottom):

• -> Bump PyTorch nightly pin past April 22, 2025 #10362

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit b8f6b50 with merge base 3066463 ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  The process '/usr/bin/sh' failed with exit code 255
• trunk / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 72ecda25335be34c8cc0a75a117412a7650908a884434d6b7592e7758adbd933 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.
ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"

[cccclai]

Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...

[swolchok]

hold on bumping?

Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...

@cccclai OK please let me know when we can do this

[cccclai]

@tugsbayasgalan is working on resolving the failure.

[swolchok]

ping @cccclai @tugsbayasgalan

[cccclai]

ping @cccclai @tugsbayasgalan

See discussion in #10319 (comment), hopefully it's resolved soon

[cccclai]

#10769 for the fix.

[swolchok]

looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22

[cccclai]

#10769 is merged as fyi

[swolchok]

test-llava-runner-linux seems to be consistently failing with SIGILL :(

[swolchok]

test-llava-runner-linux seems to be consistently failing with SIGILL :(

Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.

[swolchok]

if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.

[swolchok]

This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai

[swolchok]

in xnn_f32_vrcopysignc_ukernel__sse2_u16

the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration

[swolchok]

we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362

however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.

either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.

[digantdesai]

This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().

For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.

[swolchok]

failures seem to pattern-match to known issues on main. merging.