[Bug]: RuntimeError on RTX 5090: "no kernel image is available for execution on the device

[danhnc96]

Your current environment

Describe the bug

When running

vLLM.log

with a NVIDIA RTX 5090 GPU, I encountered the following error:

RuntimeError: CUDA error: no kernel image is available for execution on the device

From the logs, it seems that PyTorch does not support the compute capability of the RTX 5090 (sm_120):

To Reproduce

1. Use RTX 5090 GPU
2. Install vLLM with Docker or system Python environment
3. Launch the vLLM OpenAI API server
4. Engine fails to start due to CUDA kernel compatibility issue

Environment

• GPU: NVIDIA GeForce RTX 5090
• CUDA Driver Version: 12.8
• CUDA Toolkit: 12.8.93
• NVIDIA Driver: 570.124.06
• PyTorch Version: 2.x (installed via pip)
• vLLM Version: Latest (from PyPI)
• Python Version: 3.10
• OS: Ubuntu 22.04

Additional Context

It seems that the RTX 5090 uses a new compute capability (sm_120), which is currently not supported in the stable PyTorch build I'm using.

Is there a recommended way to run vLLM with this GPU? Should I:

• Switch to a nightly PyTorch build that supports sm_120?
• Build PyTorch from source with TORCH_CUDA_ARCH_LIST="12.0"?
• Wait for official support from PyTorch?

Any guidance or workaround would be greatly appreciated. Thanks!

How you are installing vllm

pip install -vvv vllm

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Patrick10203]

I have the exact same issue with the RTX 5090. I found no solution so far.

[bi1101]

You have to build the image yourself

here's the docker that you can try.

clone the vllm repo and place this docker compose right OUTSIDE the repo

vllm-Qwen3-30B-A3B-AWQ:
    container_name: vllm-Qwen3-30B-A3B-AWQ
    image: vllm-cu128
    build:
      dockerfile: docker/Dockerfile
      context: vllm
      args:
        max_jobs: 16
        RUN_WHEEL_CHECK: "false"
        nvcc_threads: 16
        torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0" # Should not be capitalized
    command: [ "--model", "cognitivecomputations/Qwen3-30B-A3B-AWQ", "--trust-remote-code", "--enable-reasoning", "--reasoning-parser", "deepseek_r1"]
    ports:
      - "8000:8000"
    environment:
      - VLLM_USE_V1=1
      - VLLM_MAX_SIZE_MB=10240
      - VLLM_FLASH_ATTN_VERSION=2
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      # - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_HUB_TOKEN}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    # runtime: nvidia
    volumes:
      - vllm_cache:/root/.cache
      - model_data:/models
    # ipc: host
    # privileged: true
    # restart: unless-stopped

To get FP8 models to work on Blackwell

After building the original image, install triton from souce to the image

FROM vllm-cu128
# Install Triton from source.
RUN pip install git+https://github.com/triton-lang/triton

[alew3]

@bi1101 after a few hours building, it failed for me when running the model

vllm-Qwen3-30B-A3B-AWQ  | ERROR 05-08 13:29:08 [core.py:412] RuntimeError: CUDA error: no kernel image is available for execution on the device
vllm-Qwen3-30B-A3B-AWQ  | ERROR 05-08 13:29:08 [core.py:412] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
vllm-Qwen3-30B-A3B-AWQ  | ERROR 05-08 13:29:08 [core.py:412] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
vllm-Qwen3-30B-A3B-AWQ  | ERROR 05-08 13:29:08 [core.py:412] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

BTW: I'm on Ubuntu 24.4 and Cuda 12.8 Nvidia Driver: 570.124.04

[bi1101]

@alew3 sorry, my mistake, the torch_cuda_arch_list: "7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0" should not be capitalized, the default arch does not include compute capability 12.0 architecture for 5090, thus the error you are experiencing.

Building took ~ 6 hours for me, I recall adding 12.0 architecture took even more time. Adding threads could help, but the build process was bottleneck at some jobs that are not multi threadable, or I'm missing sth. If you found a way to speed up building, pls do let me know.

Also, I experienced some torch compile error with FP8 model so I used --enforce-eager. After testing, AWQ models offer way better throughput compared to Fp8 so I stick with AWQ, it''s also easier to Lora finetune AWQ compared to FP8 models imo.

[cyril23]

good news: RTX 5090 + vllm: works with v0.9.0. Wait for the official docker (or build it yourself)

vllm v0.9.0 has been released 4 days ago!

• https://github.com/vllm-project/vllm/releases/tag/v0.9.0
• https://github.com/vllm-project/vllm/tree/v0.9.0

Why is that big news?

• According to https://hub.docker.com/r/vllm/vllm-openai/tags the newest official tag is still at v0.8.5.post1.
  v0.8.5.post1 is a problem because it still uses CUDA 12.4.1 - so no Blackwell support even though it uses torch==2.8.0.dev20250318+cu128
• Version v0.9.0 now uses CUDA 12.8.1 in the docker image!

If you can't wait for vllm to upload v0.9.0 to Docker Hub, you can already try it out today:

# 1. clone official vllm github repo
git clone https://github.com/vllm-project/vllm
cd vllm/
git checkout tags/v0.9.0 -b mybranch

# 2. build
DOCKER_BUILDKIT=1 sudo docker build . --target vllm-openai \
  --tag myvllm --file docker/Dockerfile --build-arg max_jobs=4 \
  --build-arg nvcc_threads=1  \
  --build-arg torch_cuda_arch_list="12.0 12.1" \
  --build-arg RUN_WHEEL_CHECK=false

# 3. run
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    myvllm --model microsoft/Phi-4-mini-instruct

So once they release v0.9.0 to Docker Hub, you will be able to just run vllm from Docker Hub without building anything:

# 1. run :)
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    vllm/vllm-openai:latest --model microsoft/Phi-4-mini-instruct

bad news: VLLM still lacks FP8 support on RTX 50xx (SM120)

clone the vllm repo and place this docker compose right OUTSIDE the repo
To get FP8 models to work on Blackwell
After building the original image, install triton from souce to the image

This does not work for me for fp8 unfortunately:

git clone https://github.com/vllm-project/vllm
cd vllm/
git checkout tags/v0.9.0 -b mybranch
sudo docker build . --target vllm-openai --tag myvllm-all \
  --file docker/Dockerfile --build-arg max_jobs=4 \
  --build-arg nvcc_threads=1  \
  --build-arg torch_cuda_arch_list="7.0 7.5 8.0 8.6 8.9 9.0+PTX 12.0 12.1" 
  --build-arg RUN_WHEEL_CHECK=false
# takes quite a long time: Building 10139.6s (42/42) FINISHED 

nano Dockerfile_all.fp8
# BEGIN Dockerfile_all.fp8
FROM myvllm-all
RUN pip install --no-cache-dir git+https://github.com/triton-lang/triton
# END Dockerfile_all.fp8

sudo docker build   -f Dockerfile_all.fp8   -t myvllm-all-fp8 .

sudo docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface     -p 8000:8000 \
  --env "HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN" \
  --env "TORCH-LOGS=+dynamo"     \
  myvllm-all-fp8 --model RedHatAI/Meta-Llama-3.1-8B-Instruct-FP8     \
  --max-model-len 8k

Same errors as usual for fp8:

INFO 05-18 08:58:46 [default_loader.py:278] Loading weights took 5.17 seconds
INFO 05-18 08:58:46 [gpu_model_runner.py:1447] Model loading took 8.4890 GiB and 6.184214 seconds
INFO 05-18 08:58:51 [backends.py:459] Using cache directory: /root/.cache/vllm/torch_compile_cache/d4491b0504/rank_0_0 for vLLM's torch.compile
INFO 05-18 08:58:51 [backends.py:469] Dynamo bytecode transform time: 4.47 s
Process EngineCore_0:
ERROR 05-18 08:58:52 [core.py:489] EngineCore failed to start.
ERROR 05-18 08:58:52 [core.py:489] Traceback (most recent call last):
ERROR 05-18 08:58:52 [core.py:489]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 480, in run_engine_core
ERROR 05-18 08:58:52 [core.py:489]     engine_core = EngineCoreProc(*args, **kwargs)
[...]
ERROR 05-18 08:58:52 [core.py:489]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/runtime/triton_heuristics.py", line 1153, in make_launcher
ERROR 05-18 08:58:52 [core.py:489]     "launch_enter_hook": binary.__class__.launch_enter_hook,
ERROR 05-18 08:58:52 [core.py:489]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-18 08:58:52 [core.py:489] torch._inductor.exc.InductorError: AttributeError: type object 'CompiledKernel' has no attribute 'launch_enter_hook'

edit: errors-details.txt added

System Info:

• x64 (AMD 9800X3D)
• 96 GB memory
• NVIDIA RTX 5090, 32 GB VRAM
• Host:
  • Windows 11 24H2
  • Driver: 576.40 from Mon May 12, 2025
  • Nvidia CUDA Toolkit 12.9
• WSL2: wsl --version -> WSL-Version: 2.4.13.0, Kernelversion: 5.15.167.4-1, WSLg-Version: 1.0.65, MSRDC-Version: 1.2.5716, Direct3D-Version: 1.611.1-81528511, DXCore-Version: 10.0.26100.1-240331-1435.ge-release, Windows-Version: 10.0.26100.3915
• Running WSL2 with Ubuntu 24 LTS, freshly installed. The only change to WSL I did is create a file "C:\Users\MYUSER.wslconfig" with content

[wsl2]
memory=85GB

Output of nvidia-smi within Ubuntu 24 WSL:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.55.01              Driver Version: 576.40         CUDA Version: 12.9     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        On  |   00000000:01:00.0  On |                  N/A |
|  0%   47C    P8             28W /  575W |    1400MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Are you sure you ran fp8 like that? According to #18153 (comment)

SM120(5080, 5090) Blackwell support still needs to be added for GEMMS to vLLM. But on the bright side CUTLASS examples and support has released for 80_blackwell_geforce_sparse_gemm and 79_blackwell_geforce_gemm. We would love some community support here.

Will try out #14452 (comment) now, looks promising: main...kaln27:vllm:main

[dingdayu]

Link:

triton-lang/triton#6805
pytorch/pytorch#153554

[voipmonitor]

@cyril23 have you managed to rum vllm with the FP8 models? I'm getting internal errors when loading Qwen/Qwen3-30B-A3B-FP8

vllm serve /root/models/Qwen/Qwen3-30B-A3B-FP8/ --enable-reasoning --reasoning-parser deepseek_r1 --max-model-len 8192 --tensor-parallel-size 2

(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] File "/root/vllm-new/vllm/vllm/_custom_ops.py", line 678, in cutlass_scaled_mm
(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] File "/root/miniconda3/envs/ktransformers-clean/lib/python3.11/site-packages/torch/_ops.py", line 1208, in call
(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=462124) ERROR 06-04 00:35:04 [multiproc_executor.py:523] RuntimeError: Error Internal

[cyril23]

@cyril23 have you managed to rum vllm with the FP8 models

@voipmonitor
I've successfully tested vLLM with FP8 models on a single RTX 5090. I haven't tried tensor parallelism since I only have one GPU.

• Use my vLLM fork: https://github.com/cyril23/vllm
  • It includes kaln27's SM120 FP8 patch from this pullrequest: [NVIDIA] Support Cutlass w8a8 FP8 for Blackwell Geforce GPUs (sm120) #17280
  • I've merged the latest main branch from https://github.com/vllm-project/vllm into my fork to include recent PyTorch and CUDA updates.
• Full discussion: [Installation]: Dual 5090's (sm120, cu128) Issues Running vLLM #16515 (comment)

# Clone the repository (includes kaln27's FP8 patch and more recent commits)
git clone https://github.com/cyril23/vllm vllm-kaln27-updated
cd vllm-kaln27-updated/

# Build:
DOCKER_BUILDKIT=1 sudo docker build . \
  --target vllm-openai \
  --tag myvllm-kaln27-updated \
  --file docker/Dockerfile \
  --build-arg max_jobs=4 --build-arg nvcc_threads=1  \
  --build-arg torch_cuda_arch_list="12.0 12.1" \
  --build-arg RUN_WHEEL_CHECK=false

# Run:
export HUGGING_FACE_HUB_TOKEN=YOUR_TOKEN # only needed for private or restricted models
sudo docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN" \
    myvllm-kaln27-updated \
    --model RedHatAI/Qwen3-30B-A3B-FP8-dynamic \
    --gpu-memory-utilization 0.9 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --enable-reasoning --reasoning-parser deepseek_r1 \
    --max-model-len 8192 \
    --tensor-parallel-size 2

# edit: added tensor parallelism arg. edit: fixed clone command

Does that work for you?

By the way, if you’re interested, you can also try NVIDIA's TensorRT-LLM:

• Unlike kaln27’s vLLM PR (which only enables basic FP8 inference), TensorRT-LLM v0.19.0 adds working kernels for FP8 K/V caching on Blackwell (SM 120) GPUs like the RTX 5090.
• In high‐QPS scenarios, TensorRT-LLM can achieve even higher FP8 throughput than vLLM (even without FP8 K/V cache). This might be because kaln27's FP8 implementation in vLLM still has some room for optimization.
• Full instructions are here: tensorrt-llm 0.19.0 requires torch<=2.7.0a0,>=2.6.0 (and therefore does not install the current 2.7.0 but the old 2.6.0 which makes it Blackwell incompatible) NVIDIA/TensorRT-LLM#4275 (comment)

[voipmonitor]

@cyril23 have you managed to rum vllm with the FP8 models

@voipmonitor I've successfully tested vLLM with FP8 models on a single RTX 5090. I haven't tried tensor parallelism since I only have one GPU.

Thank you. I have tried your branch and now I'm able to run FP8 model RedHatAI/Qwen3-30B-A3B-FP8-dynamic though I'm not able to run Qwen/Qwen3-30B-A3B-FP8 - have you tried the Qwen3-30B-A3B-FP8 ? It fails with RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell - it seems like torch does not implement this yet? (or what component it might be?

(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]     torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]   File "/root/miniconda3/envs/vllm-clean2/lib/python3.11/site-packages/torch/_ops.py", line 1208, in __call__
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]     return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorker rank=0 pid=722847) ERROR 06-05 14:28:26 [multiproc_executor.py:522] RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell

Regarding the tensor parallelism - I have just tested that if I install vllm and pytorch "pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128" and compile vllm directly on the host (ubuntu 24 lts) with Cuda compilation tools, release 12.8, V12.8.93 - it does not crash. I will try to figure out what differs in all the docker images which crashes the parallelism

I'm also curious why when compiling vllm it can use all my 36 cores when nvidia compiler starts while on the docker image it can compile only with 4 parallel tasks (which takes ages) and I have tried almost everything in the docker - injecting -j 36 to the ninja and setting all kind of parameters which should do it - this is annoying when building docker images with vllm

TensorRT-LLM - not yet. I'm new to all of this progressing slowly - but I will try it.

I have also tried Ktransformer with hybrid CPU/GPU configuration and the recent AMX implementation getting interesting results

[cyril23]

though I'm not able to run Qwen/Qwen3-30B-A3B-FP8 - have you tried the Qwen3-30B-A3B-FP8 ? It fails with RuntimeError: Currently, block scaled fp8 gemm is not implemented for Blackwell - it seems like torch does not implement this yet?+

Yes, it fails with the same message for me. The message comes directly from kaln27's Pullrequest which I've integrated into my branch. See https://github.com/vllm-project/vllm/pull/17280/files - this part. This type is just not implemented yet.

Regarding the tensor parallelism - I have just tested that if I install vllm and pytorch "pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128" and compile vllm directly on the host (ubuntu 24 lts) with Cuda compilation tools, release 12.8, V12.8.93 - it does not crash.

The current vLLM Dockerfile uses pytorch stable (for non-arm architecture). So apparently pytorch 2.7.0 is too old and still contains some bugs for RTX Blackwell? But good to know you can at least work-around it for the moment like that.