PyTorch 2.0 nightly: compile does not support capabilities sm_89 (RTX 4090 and others)
#95082
Comments
seemethere
added
module: cuda
|
@ptillet this looks like it is coming from Triton. Does Triton support sm_89? |
|
It should support |
|
|
|
@pcuenca can you try upgrading Triton and see if that fixes this? @malfet is there an issue with dependencies causing the wrong version of Triton to get installed along with our nightlies? Perhaps we need some sort of version check that issues a warning if users don't have the proper Triton installed. |
|
@jansel I get multiple messages like this after upgrading Triton: |
|
I've seen similar error messages raised by LLVM from here, but don't know why |
|
I'm having the same sort of problem on an RTX 6000 Ada when trying to use torch.compile in pytorch 2.0. The exact error I get is: From reading around elsewhere I was under the impression that this change in triton is required to fix it? |
|
Triton uses |
|
I get the same repeated messages as @pcuenca after pip install triton==2.0.0.a2 ('sm_89' is not a recognized processor for this target (ignoring processor)). I also am not seeing training progress. I should note, that I still have 'pytorch-triton==2.0.0+c8bfe3f548' installed after installing 'triton==2.0.0.a2'. Removing pytorch-triton causes the error "torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised ImportError: cannot import name 'next_power_of_2' from 'triton' (unknown location)" to be thrown instead. |
|
I think there might be an issue with Can you try:
|
|
Tried uninstalling both and reinstalling triton==2.0.0.a2. First I had to suppress some dependency issues when running pip since torch requires pytorch-triton to be installed. When I got it installed I got a much longer pause than I did the first time, but it eventually started spitting out a similar message, interestingly it looks like some weird nesting is happening? then it spit out tons of the same message as before: Finally it spit out a bunch of python code that looked like it was being converted by the compiler: It goes on for a few hundred more lines with similar tmp[0-9]* = tl.load(...) lines. |
|
For whom it might be helpful I was just able to successfully compile a random model on A100 by following |
gchanan
added
the
triaged
|
@weiwangmeta I uninstalled There is no These are the versions that got installed: (I'll try later to see if a more recent wheel becomes available) |
Should fix pytorch#95082 This hash commit is supposed to fix sm_89 issue. Pull Request resolved: pytorch#95247 Approved by: https://github.com/ngimel, https://github.com/seemethere
Should fix #95082 This hash commit is supposed to fix sm_89 issue. Pull Request resolved: #95247 Approved by: https://github.com/ngimel, https://github.com/seemethere Co-authored-by: Wei Wang <[email protected]>
We are having some nightly build issues (the latest nightly was 2 days ago, so torch does not get updated and keeps calling for the broken wheel). Please use the three steps in this comment: Please let us know if this still does not work for you. |
Not OP but I had the same issue and your workaround was successful for me. Thanks! |
We have fixed 2.0.0-rc1 with 2.0.0-rc2, now just the step 1 would work fine too. pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/test/cu117 |
|
@weiwangmeta I can confirm the latest RC works fine. Thanks a lot! |
|
@pcuenca Thanks for the confirmation! Since you originally tried nightly version, just fyi that the nightly binaries were just available and should work fine as well: Collecting torch |
Should fix pytorch#95082 This hash commit is supposed to fix sm_89 issue. Pull Request resolved: pytorch#95247 Approved by: https://github.com/ngimel, https://github.com/seemethere Co-authored-by: Wei Wang <[email protected]>
Should fix pytorch#95082 This hash commit is supposed to fix sm_89 issue. Pull Request resolved: pytorch#95247 Approved by: https://github.com/ngimel, https://github.com/seemethere
Uh oh!
There was an error while loading. Please reload this page.
🐛 Describe the bug
Trying the nightlies today (20230217) on RTX 4090 I get the following error when attempting to use
compile():As mentioned in #95081, the resolved nightly is
2.0.0.dev20230213+cu118.Repeating the same test with the cuda 11.7 wheels, the behavior is identical.
Error logs
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 549, in _worker_compile
kernel.precompile(warm_cache_only_with_cc=cc)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 69, in precompile
self.launchers = [
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 70, in
self._precompile_config(c, warm_cache_only_with_cc)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/triton_ops/autotune.py", line 83, in _precompile_config
triton.compile(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/triton/compiler.py", line 1256, in compile
asm, shared, kernel_name = _compile(fn, signature, device, constants, configs[0], num_warps, num_stages,
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/triton/compiler.py", line 901, in _compile
name, asm, shared_mem = _triton.code_gen.compile_ttir(backend, module, device, num_warps, num_stages, extern_libs, cc)
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error : PTX .version 7.4 does not support .target sm_89
ptxas fatal : Ptx assembly aborted due to errors
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/dynamo/output_graph.py", line 670, in call_user_compiler
compiled_fn = compiler_fn(gm, self.fake_example_inputs())
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/dynamo/debug_utils.py", line 1055, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/init.py", line 1374, in call
return self.compile_fn(model, inputs, config_patches=self.config)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 453, in compile_fx
return aot_autograd(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/backends/common.py", line 48, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 2481, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 2178, in create_aot_dispatcher_function
compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1412, in aot_wrapper_dedupe
return compiler_fn(flat_fn, leaf_flat_args, aot_config)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_functorch/aot_autograd.py", line 1062, in aot_dispatch_base
compiled_fw = aot_config.fw_compiler(fw_module, flat_args)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 428, in fw_compiler
return inner_compile(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/debug_utils.py", line 595, in debug_wrapper
compiled_fn = compiler_fn(gm, example_inputs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/debug.py", line 239, in inner
return fn(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/compile_fx.py", line 177, in compile_fx_inner
compiled_fn = graph.compile_to_fn()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/graph.py", line 586, in compile_to_fn
return self.compile_to_module().call
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/graph.py", line 575, in compile_to_module
mod = PyCodeCache.load(code)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 528, in load
exec(code, mod.dict, mod.dict)
File "/tmp/torchinductor_pedro/b4/cb4wgmozs4bqzsoemk2mnsij7oypadaj3znkbzxbgpyxre3st3cy.py", line 104, in
async_compile.wait(globals())
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 715, in wait
scope[key] = result.result()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_inductor/codecache.py", line 573, in result
self.future.result()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/_base.py", line 446, in result
return self.__get_result()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error : PTX .version 7.4 does not support .target sm_89
ptxas fatal : Ptx assembly aborted due to errors
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/pedro/code/bench/benchmark_diffusers_pt2.py", line 47, in
do_tests(bs)
File "/home/pedro/code/bench/benchmark_diffusers_pt2.py", line 38, in do_tests
pipe(prompt, num_inference_steps=steps, num_images_per_prompt=batch_size).images
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/pedro/code/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 643, in call
noise_pred = self.unet(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
return fn(*args, **kwargs)
File "/home/pedro/code/diffusers/src/diffusers/models/unet_2d_condition.py", line 554, in forward
t_emb = self.time_proj(timesteps)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 330, in catch_errors
return callback(frame, cache_size, hooks)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
result = inner_convert(frame, cache_size, hooks)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
return fn(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
return _compile(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 324, in _compile
out_code = transform_code_object(code, transform)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 376, in transform_code_object
transformations(instructions, code_options)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 311, in transform
tracer.run()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1681, in run
super().run()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 569, in run
and self.step()
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 532, in step
getattr(self, inst.opname)(inst)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1747, in RETURN_VALUE
self.output.compile_subgraph(
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 517, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 588, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
r = func(*args, **kwargs)
File "/home/pedro/miniconda/envs/hf/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 675, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/filelSnH8K, line 6; error : PTX .version 7.4 does not support .target sm_89
ptxas fatal : Ptx assembly aborted due to errors
You can suppress this exception and fall back to eager by setting:
torch._dynamo.config.suppress_errors = True
Minified repro
No response
Versions
PyTorch version: 2.0.0.dev20230213+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.35
Python version: 3.9.16 (main, Jan 11 2023, 16:05:54) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090
Nvidia driver version: 525.85.12
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7950X 16-Core Processor
CPU family: 25
Model: 97
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
Stepping: 2
Frequency boost: enabled
CPU max MHz: 5879.8818
CPU min MHz: 3000.0000
BogoMIPS: 9000.38
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization: AMD-V
L1d cache: 512 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 16 MiB (16 instances)
L3 cache: 64 MiB (2 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20230213+cu118
[pip3] torchvision==0.15.0.dev20230217+cu118
[conda] numpy 1.24.1 pypi_0 pypi
[conda] pytorch-triton 2.0.0+0d7e753227 pypi_0 pypi
[conda] torch 2.0.0.dev20230213+cu118 pypi_0 pypi
[conda] torchvision 0.15.0.dev20230217+cu118 pypi_0 pypi
cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire
The text was updated successfully, but these errors were encountered: