[Bug]: Cutlass MoE for Llama 4 FP8 broken in 0.9.0 ?

[jmkuebler]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Amazon Linux 2023.7.20250428 (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Clang version                : Could not collect
CMake version                : version 3.26.4
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu126
Is debug build               : False
CUDA used to build PyTorch   : 12.6
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.1.134-150.224.amzn2023.x86_64-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.6.85
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 570.133.20
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        48 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               192
On-line CPU(s) list:                  0-191
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 7R13 Processor
CPU family:                           25
Model:                                1
Thread(s) per core:                   2
Core(s) per socket:                   48
Socket(s):                            2
Stepping:                             1
BogoMIPS:                             5299.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            3 MiB (96 instances)
L1i cache:                            3 MiB (96 instances)
L2 cache:                             48 MiB (96 instances)
L3 cache:                             384 MiB (12 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-47,96-143
NUMA node1 CPU(s):                    48-95,144-191
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Mitigation; safe RET
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0
[pip3] torchaudio==2.7.0
[pip3] torchvision==0.22.0
[pip3] transformers==4.51.3
[pip3] triton==3.3.0
[conda] numpy                     2.2.6                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.6.4.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.6.80                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.6.77                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.6.77                  pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.5.1.17                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.3.0.4                 pypi_0    pypi
[conda] nvidia-cufile-cu12        1.11.1.6                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.7.77                pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.7.1.2                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.5.4.2                 pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.6.3                    pypi_0    pypi
[conda] nvidia-nccl-cu12          2.26.2                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.85                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.6.77                  pypi_0    pypi
[conda] pyzmq                     26.4.0                   pypi_0    pypi
[conda] torch                     2.7.0                    pypi_0    pypi
[conda] torchaudio                2.7.0                    pypi_0    pypi
[conda] torchvision               0.22.0                   pypi_0    pypi
[conda] transformers              4.51.3                   pypi_0    pypi
[conda] triton                    3.3.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.9.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      48-95,144-191   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib:/lib:/opt/amazon/efa/lib64:/opt/amazon/openmpi/lib64:/opt/amazon/ofi-nccl/lib64:/usr/local/cuda/lib:/usr/local/cuda:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib:/usr/lib:/lib
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

On vllm 0.9.0 and also the recent nightly version, there is a functional break when running the FP8 variant of Llama-4 Maverick. I guess it might be a bug in the Cutlass MoE kernel?

model=meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
lm_eval --model vllm \
        --model_args "pretrained=$model,tensor_parallel_size=8,max_model_len=100000" \
        --tasks  gsm8k \
        --batch_size auto \
        --seed 42 \
        --trust_remote_code \
        --limit 100

on 0.8.5.post1 above results in 94.00% accuracy, whilst on 0.9.0 the accuracy drops to 31.00%.
The problem persists in the nightly built of 0.9.1.dev9+gd6c86d09a

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jmkuebler]

Additionally, on the non-FP8 variant, the accuracy is fine.

[jmkuebler]

tagging @mgoin @tlrmchlsmth, as you recently reviewed Cutlass MoE PRs.

[tlrmchlsmth]

Some updates:

• I can repro this with model=RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic
• The issue doesn't repro with --enforce-eager

[tlrmchlsmth]

Narrowed the problem down to #15956

[mgoin]

Resolved by #18440

Thank you very much for finding this @jmkuebler.

[jmkuebler]

Thanks a lot @mgoin @bnellnm for the quick fix. However, for me the model now breaks at compilation. Did it successfully run through on your end?
I used my command above

model=meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8
lm_eval --model vllm \
        --model_args "pretrained=$model,tensor_parallel_size=8,max_model_len=100000" \
        --tasks  gsm8k \
        --batch_size auto \
        --seed 42 \
        --trust_remote_code \
        --limit 100

The compilation error is introduced by #18440

on the commit before the "fix" the problem is as above, i.e., the model runs, but gives bad accuracy

export VLLM_COMMIT=23baa2180b0ebba5ae94073ba9b8e93f88b75486
pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

on the commit of the fix the model does not compile

export VLLM_COMMIT=92247c522e216f9d010db1c648dc783dbf141704 
pip install https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

Partial output of model launch

INFO 05-21 13:09:46 [core.py:65] Initializing a V1 LLM engine (v0.9.1.dev27+geca18691d) with config: model='meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8', speculative_config=None, tokenizer='meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=100000, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=1234, served_model_name=meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level": 3, "custom_ops": ["none"], "splitting_ops": ["vllm.unified_attention", "vllm.unified_attention_with_output"], "compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "use_cudagraph": true, "cudagraph_num_of_warmups": 1, "cudagraph_capture_sizes": [512, 504, 496, 488, 480, 472, 464, 456, 448, 440, 432, 424, 416, 408, 400, 392, 384, 376, 368, 360, 352, 344, 336, 328, 320, 312, 304, 296, 288, 280, 272, 264, 256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 512}
WARNING 05-21 13:09:46 [multiproc_worker_utils.py:306] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3, 4, 5, 6, 7], buffer_handle=(8, 10485760, 10, 'psm_1dc4cb91'), local_subscribe_addr='ipc:///tmp/858144af-c22b-45f1-86f5-50f0db7d548a', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f606f50>
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_4083347f'), local_subscribe_addr='ipc:///tmp/19207bb1-d13f-432a-be22-db2e2be10c0a', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f604610>
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_bdf8cbdb'), local_subscribe_addr='ipc:///tmp/a40abf4c-29b7-46b2-aeb6-bc5aeb0cf468', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f605270>
(VllmWorker rank=2 pid=755138) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_811415ee'), local_subscribe_addr='ipc:///tmp/39636d39-f6c0-4906-9c45-04036a1765b8', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f607310>
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_e3d1214d'), local_subscribe_addr='ipc:///tmp/26ae8666-806c-4c41-886b-1f978862d430', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f6061a0>
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_d97bb56a'), local_subscribe_addr='ipc:///tmp/bb511a92-8724-4933-9acc-7da73948f362', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f604c70>
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_ea336ad9'), local_subscribe_addr='ipc:///tmp/77cdcf3e-6af5-4ea6-b9e5-77c3cc93335e', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f6044f0>
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_0b8f3c08'), local_subscribe_addr='ipc:///tmp/d8c7e955-5fb0-43c8-a9e0-1d4d7eaf770d', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 05-21 13:09:46 [utils.py:2664] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fe55f605300>
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:46 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_2b47eb43'), local_subscribe_addr='ipc:///tmp/62ff5b6f-a64e-41df-b939-81039bfd289d', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=2 pid=755138) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=2 pid=755138) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:50 [utils.py:1071] Found nccl from library libnccl.so.2
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:50 [pynccl.py:69] vLLM is using nccl==2.26.2
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=2 pid=755138) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:52 [custom_all_reduce_utils.py:245] reading GPU P2P access cache from /home/ec2-user/.cache/vllm/gpu_p2p_access_cache_for_0,1,2,3,4,5,6,7.json
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:52 [shm_broadcast.py:266] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_472d373c'), local_subscribe_addr='ipc:///tmp/1d1a3b09-2a40-4ca1-89a2-3b7ca7630ce9', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker rank=2 pid=755138) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:53 [parallel_state.py:1079] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
(VllmWorker rank=0 pid=755136) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=1 pid=755137) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=0 pid=755136) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=1 pid=755137) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=4 pid=755140) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=7 pid=755143) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=4 pid=755140) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=7 pid=755143) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=5 pid=755141) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=6 pid=755142) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=5 pid=755141) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=6 pid=755142) INFO 05-21 13:09:59 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=3 pid=755139) WARNING 05-21 13:09:59 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=3 pid=755139) INFO 05-21 13:09:59 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:00 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=2 pid=755138) WARNING 05-21 13:10:00 [topk_topp_sampler.py:58] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:00 [gpu_model_runner.py:1503] Starting to load model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8...
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:00 [cuda.py:216] Using Flash Attention backend on V1 engine.
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:00 [backends.py:37] Using InductorAdaptor
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:01 [weight_utils.py:307] Time spent downloading weights for meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8: 0.672522 seconds
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:01 [weight_utils.py:291] Using model weights format ['*.safetensors']
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:01 [weight_utils.py:291] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/84 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   4% Completed | 3/84 [00:00<00:05, 15.29it/s]
Loading safetensors checkpoint shards:   6% Completed | 5/84 [00:00<00:04, 15.92it/s]
Loading safetensors checkpoint shards:   8% Completed | 7/84 [00:00<00:05, 14.24it/s]
Loading safetensors checkpoint shards:  11% Completed | 9/84 [00:00<00:04, 15.93it/s]
Loading safetensors checkpoint shards:  14% Completed | 12/84 [00:00<00:04, 17.66it/s]
Loading safetensors checkpoint shards:  18% Completed | 15/84 [00:00<00:03, 18.45it/s]
Loading safetensors checkpoint shards:  20% Completed | 17/84 [00:01<00:04, 14.72it/s]
Loading safetensors checkpoint shards:  23% Completed | 19/84 [00:01<00:04, 15.71it/s]
Loading safetensors checkpoint shards:  25% Completed | 21/84 [00:01<00:04, 14.65it/s]
Loading safetensors checkpoint shards:  27% Completed | 23/84 [00:01<00:04, 14.09it/s]
Loading safetensors checkpoint shards:  30% Completed | 25/84 [00:01<00:03, 14.88it/s]
Loading safetensors checkpoint shards:  32% Completed | 27/84 [00:01<00:04, 14.19it/s]
Loading safetensors checkpoint shards:  36% Completed | 30/84 [00:01<00:03, 14.21it/s]
Loading safetensors checkpoint shards:  39% Completed | 33/84 [00:02<00:03, 16.02it/s]
Loading safetensors checkpoint shards:  42% Completed | 35/84 [00:02<00:02, 16.77it/s]
Loading safetensors checkpoint shards:  44% Completed | 37/84 [00:02<00:03, 15.44it/s]
Loading safetensors checkpoint shards:  46% Completed | 39/84 [00:02<00:02, 15.29it/s]
Loading safetensors checkpoint shards:  49% Completed | 41/84 [00:02<00:02, 16.16it/s]
Loading safetensors checkpoint shards:  51% Completed | 43/84 [00:02<00:03, 12.99it/s]
Loading safetensors checkpoint shards:  54% Completed | 45/84 [00:02<00:02, 13.64it/s]
Loading safetensors checkpoint shards:  56% Completed | 47/84 [00:03<00:02, 14.82it/s]
Loading safetensors checkpoint shards:  58% Completed | 49/84 [00:03<00:02, 13.85it/s]
Loading safetensors checkpoint shards:  61% Completed | 51/84 [00:03<00:02, 15.01it/s]
Loading safetensors checkpoint shards:  63% Completed | 53/84 [00:03<00:02, 13.80it/s]
Loading safetensors checkpoint shards:  65% Completed | 55/84 [00:03<00:02, 13.08it/s]
Loading safetensors checkpoint shards:  68% Completed | 57/84 [00:03<00:01, 13.80it/s]
Loading safetensors checkpoint shards:  70% Completed | 59/84 [00:04<00:02, 11.81it/s]
Loading safetensors checkpoint shards:  73% Completed | 61/84 [00:04<00:01, 13.29it/s]
Loading safetensors checkpoint shards:  75% Completed | 63/84 [00:04<00:01, 14.73it/s]
Loading safetensors checkpoint shards:  77% Completed | 65/84 [00:04<00:01, 15.85it/s]
Loading safetensors checkpoint shards:  80% Completed | 67/84 [00:04<00:01, 14.82it/s]
Loading safetensors checkpoint shards:  82% Completed | 69/84 [00:04<00:00, 15.64it/s]
Loading safetensors checkpoint shards:  85% Completed | 71/84 [00:04<00:00, 15.49it/s]
Loading safetensors checkpoint shards:  87% Completed | 73/84 [00:04<00:00, 14.27it/s]
Loading safetensors checkpoint shards:  89% Completed | 75/84 [00:05<00:00, 13.85it/s]
Loading safetensors checkpoint shards:  93% Completed | 78/84 [00:05<00:00, 15.90it/s]
Loading safetensors checkpoint shards:  95% Completed | 80/84 [00:05<00:00, 14.47it/s]
Loading safetensors checkpoint shards:  98% Completed | 82/84 [00:05<00:00, 15.56it/s]
Loading safetensors checkpoint shards: 100% Completed | 84/84 [00:05<00:00, 14.46it/s]
Loading safetensors checkpoint shards: 100% Completed | 84/84 [00:05<00:00, 14.78it/s]
(VllmWorker rank=0 pid=755136) 
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:31 [default_loader.py:279] Loading weights took 28.97 seconds
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:31 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 31.960905 seconds
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:32 [default_loader.py:279] Loading weights took 28.72 seconds
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:32 [default_loader.py:279] Loading weights took 28.59 seconds
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:32 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 32.431910 seconds
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:32 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 32.335586 seconds
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:32 [default_loader.py:279] Loading weights took 30.49 seconds
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:32 [default_loader.py:279] Loading weights took 29.38 seconds
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:32 [default_loader.py:279] Loading weights took 28.35 seconds
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:33 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 33.255892 seconds
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:33 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 33.346318 seconds
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:33 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 33.257356 seconds
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:33 [default_loader.py:279] Loading weights took 29.26 seconds
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:33 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 33.919588 seconds
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:36 [default_loader.py:279] Loading weights took 31.57 seconds
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:36 [gpu_model_runner.py:1521] Model loading took 48.8682 GiB and 36.341125 seconds
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:36 [gpu_model_runner.py:1823] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 7 image items of the maximum feature size.
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:48 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_3_0 for vLLM's torch.compile
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:48 [backends.py:469] Dynamo bytecode transform time: 10.37 s
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:48 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_5_0 for vLLM's torch.compile
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:48 [backends.py:469] Dynamo bytecode transform time: 10.40 s
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:48 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_7_0 for vLLM's torch.compile
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:48 [backends.py:469] Dynamo bytecode transform time: 10.55 s
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:48 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_0_0 for vLLM's torch.compile
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:48 [backends.py:469] Dynamo bytecode transform time: 10.66 s
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:49 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_1_0 for vLLM's torch.compile
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:49 [backends.py:469] Dynamo bytecode transform time: 10.76 s
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:49 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_2_0 for vLLM's torch.compile
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:49 [backends.py:469] Dynamo bytecode transform time: 10.81 s
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:49 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_4_0 for vLLM's torch.compile
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:49 [backends.py:469] Dynamo bytecode transform time: 10.86 s
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:49 [backends.py:459] Using cache directory: /home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_6_0 for vLLM's torch.compile
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:49 [backends.py:469] Dynamo bytecode transform time: 11.40 s
(VllmWorker rank=3 pid=755139) INFO 05-21 13:10:51 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=5 pid=755141) INFO 05-21 13:10:51 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=0 pid=755136) INFO 05-21 13:10:51 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=7 pid=755143) INFO 05-21 13:10:52 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=2 pid=755138) INFO 05-21 13:10:52 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=1 pid=755137) INFO 05-21 13:10:52 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=4 pid=755140) INFO 05-21 13:10:52 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=6 pid=755142) INFO 05-21 13:10:52 [backends.py:160] Cache the graph of shape None for later use
(VllmWorker rank=3 pid=755139) INFO 05-21 13:11:33 [backends.py:172] Compiling a graph for general shape takes 44.55 s
(VllmWorker rank=0 pid=755136) INFO 05-21 13:11:34 [backends.py:172] Compiling a graph for general shape takes 45.14 s
(VllmWorker rank=2 pid=755138) INFO 05-21 13:11:35 [backends.py:172] Compiling a graph for general shape takes 45.33 s
(VllmWorker rank=7 pid=755143) INFO 05-21 13:11:35 [backends.py:172] Compiling a graph for general shape takes 46.25 s
(VllmWorker rank=1 pid=755137) INFO 05-21 13:11:35 [backends.py:172] Compiling a graph for general shape takes 46.31 s
(VllmWorker rank=6 pid=755142) INFO 05-21 13:11:36 [backends.py:172] Compiling a graph for general shape takes 45.75 s
(VllmWorker rank=4 pid=755140) INFO 05-21 13:11:36 [backends.py:172] Compiling a graph for general shape takes 46.36 s
(VllmWorker rank=5 pid=755141) INFO 05-21 13:11:37 [backends.py:172] Compiling a graph for general shape takes 48.19 s
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 360, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     def forward(
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 360, in forward
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     def forward(
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     raise e
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     raise e
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "<eval_with_key>.98", line 526, in forward
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "<eval_with_key>.98", line 526, in forward
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     submod_4 = self.submod_4(getitem_7, s0, l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_, getitem_8, l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2, l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_7 = l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_8 = l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2 = l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_ = None
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     submod_4 = self.submod_4(getitem_7, s0, l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_, getitem_8, l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2, l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_7 = l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_8 = l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2 = l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_ = None
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/backends.py", line 651, in __call__
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/backends.py", line 651, in __call__
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.compiled_graph_for_general_shape(*args)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.compiled_graph_for_general_shape(*args)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1201, in forward
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return compiled_fn(full_args)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return compiled_fn(full_args)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 328, in runtime_wrapper
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     all_outs = call_func_at_runtime_with_args(
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     all_outs = call_func_at_runtime_with_args(
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     out = normalize_as_list(f(args))
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     out = normalize_as_list(f(args))
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 360, in forward
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 689, in inner_fn
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outs = compiled_fn(args)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     def forward(
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outs = compiled_fn(args)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 495, in wrapper
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return compiled_fn(runtime_args)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return compiled_fn(runtime_args)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 460, in __call__
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 460, in __call__
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.current_callable(inputs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.current_callable(inputs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2404, in run
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 2404, in run
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 185, in determine_available_memory
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return model(new_inputs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return model(new_inputs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "/home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_3_0/inductor_cache/zw/czws3oske7dcjwct2h736pe4ltyqymjnvfgrn7tfkhgja7fyhf4e.py", line 894, in call
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.model_runner.profile_run()
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "/home/ec2-user/.cache/vllm/torch_compile_cache/3784573129/rank_0_0/inductor_cache/77/c7755hx6i344r4atod2clcop7h2hvya32l5j2hexnevugeyufrpn.py", line 894, in call
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     triton_poi_fused_view_4.run(buf27, triton_poi_fused_view_4_xnumel, stream=stream3)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1854, in profile_run
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     triton_poi_fused_view_4.run(buf27, triton_poi_fused_view_4_xnumel, stream=stream0)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 909, in run
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     hidden_states = self._dummy_run(self.max_num_tokens)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 909, in run
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.autotune_to_one_config(*args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     raise e
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     self.autotune_to_one_config(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 763, in autotune_to_one_config
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 360, in forward
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in __call__
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 763, in autotune_to_one_config
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     timings = self.benchmark_all_configs(*args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     def forward(
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1695, in _dummy_run
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     timings = self.benchmark_all_configs(*args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 737, in benchmark_all_configs
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     outputs = model(
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 737, in benchmark_all_configs
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     timings = {
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     timings = {
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 738, in <dictcomp>
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 738, in <dictcomp>
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     launcher: self.bench(launcher, *args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     launcher: self.bench(launcher, *args, **kwargs)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 616, in bench
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "<eval_with_key>.98", line 526, in forward
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 616, in bench
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return benchmarker.benchmark_gpu(kernel_call, rep=40)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/llama.py", line 573, in forward
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(*args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py", line 768, in forward
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     submod_4 = self.submod_4(getitem_7, s0, l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_, getitem_8, l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2, l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2, l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_, l_positions_, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_);  getitem_7 = l_self_modules_layers_modules_1_modules_self_attn_modules_o_proj_parameters_weight_ = getitem_8 = l_self_modules_layers_modules_1_modules_post_attention_layernorm_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_router_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_gate_up_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_shared_expert_modules_down_proj_parameters_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w13_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides1 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_parameters_w2_weight_scale_ = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_ab_strides2 = l_self_modules_layers_modules_1_modules_feed_forward_modules_experts_quant_method_c_strides2 = l_self_modules_layers_modules_2_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_2_modules_self_attn_modules_qkv_proj_parameters_weight_ = None
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return benchmarker.benchmark_gpu(kernel_call, rep=40)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/backends.py", line 651, in __call__
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 39, in wrapper
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     model_output = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.language_model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 39, in wrapper
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self.compiled_graph_for_general_shape(*args)
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(self, *args, **kwargs)
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 238, in __call__
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._wrapped_call(self, *args, **kwargs)
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return fn(self, *args, **kwargs)
(VllmWorker rank=2 pid=755138) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
(VllmWorker rank=3 pid=755139) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 243, in benchmark_gpu
(VllmWorker rank=6 pid=755142) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 655, in _fn
(VllmWorker rank=4 pid=755140) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     output = self.compiled_callable(*args, **kwargs)
(VllmWorker rank=7 pid=755143) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in __call__
(VllmWorker rank=1 pid=755137) ERROR 05-21 13:11:53 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=5 pid=755141) ERROR 05-21 13:11:53 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=755136) ERROR 05-21 13:11:53 [multiproc_executor.py:522]   File "venv/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py", line 243, in benchmark_gpu 

[tlrmchlsmth]

@jmkuebler I'm pretty sure you have either stale binaries or there's an issue with your torch.compile cache (you can clear that by rm -r ~/.cache/vllm/torch_compile_cache/)

Thanks again for finding this one!

[jmkuebler]

@tlrmchlsmth
the error persist for me even after emptying the torch.compile cache. Also tried in a fresh venv just installing the whl from the commit, but same issue.

as said, when I use the whl from one commit earlier, the compilation runs. I don't know how to test otherwise whether I have stale binaries.

maybe you can run a quick check with RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic on your end just to be sure.

[bnellnm]

@tlrmchlsmth the error persist for me even after emptying the torch.compile cache. Also tried in a fresh venv just installing the whl from the commit, but same issue.

as said, when I use the whl from one commit earlier, the compilation runs. I don't know how to test otherwise whether I have stale binaries.

maybe you can run a quick check with RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic on your end just to be sure.

@jmkuebler both models are working for me. You may want to try also deleting any /tmp/torchinductor_$USER directory (if it exists).

[tlrmchlsmth]

Confirming: The fix works on my end too