[Bug]: [V1][gpu_model_runner.py] CUDA memory error

[rajagond]

Your current environment

I am using vllm docker image: docker pull vllm/vllm-openai:v0.9.1rc1

🐛 Describe the bug

Hi,

I am making some changes to vllm/v1/worker/gpu_model_runner.py and trying to initialize some custom tensors inside the GPUModelRunner class. Specifically, I have added the following code:

self.query_first_loc = torch.zeros(self.max_num_reqs + 1,
                                   dtype=torch.int32,
                                   device=self.device)
self.query_second_loc = torch.zeros(self.max_num_reqs + 1,
                                    dtype=torch.int32,
                                    device=self.device)
self.seq_lens_first = torch.zeros(self.max_num_reqs,
                                  dtype=torch.int32,
                                  device=self.device)
self.seq_lens_second = torch.zeros(self.max_num_reqs,
                                   dtype=torch.int32,
                                   device=self.device)

However, I am encountering the following error at runtime:

(VllmWorker rank=0 pid=76227) 
[rank2]:[W610 03:20:31.540991085 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=176, addr=[localhost]:45008, remote=[localhost]:60771): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7fff545785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7fff3863cafe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7fff3863ee40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7fff3863f74a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7fff386391a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7ffee4299989 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7ffff5079253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7ffff7ceaac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7ffff7d7c850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank2]:[W610 03:20:31.545487967 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 2] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes
[rank3]:[W610 03:20:31.546034582 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=179, addr=[localhost]:45022, remote=[localhost]:60771): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7fff545785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7fff3863cafe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7fff3863ee40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7fff3863f74a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7fff386391a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7ffee4299989 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7ffff5079253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7ffff7ceaac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7ffff7d7c850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank3]:[W610 03:20:31.550886481 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 3] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes
[rank4]:[W610 03:20:31.564445761 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=182, addr=[localhost]:45036, remote=[localhost]:60771): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7fff545785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7fff3863cafe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7fff3863ee40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7fff3863f74a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7fff386391a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7ffee4299989 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7ffff5079253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7ffff7ceaac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7ffff7d7c850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank4]:[W610 03:20:31.568616014 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes
[rank5]:[W610 03:20:31.628674019 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=185, addr=[localhost]:45052, remote=[localhost]:60771): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7fff545785e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7fff3863cafe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7fff3863ee40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7fff3863f74a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7fff386391a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7ffee4299989 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7ffff5079253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7ffff7ceaac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7ffff7d7c850 in /lib/x86_64-linux-gnu/libc.so.6)

[rank5]:[W610 03:20:31.632436958 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes

I'm not sure why this is happening.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tshak]

We're seeing this frequently on vLLM server v0.9.1 running llama3.3-70b

[rajagond]

If you'd like, you can try TokenWeave; it's built on top of vLLM v0.8.5 and doesn't currently support CUDA Graphs, but it still provides significant gains over vLLM v0.9.1 for Llama-3.3-70B on 8×H100.

[mgoin]

Could you share the full patch you have made to the vLLM source code? Does it work fine without your additional changes?

[rajagond]

Could you share the full patch you have made to the vLLM source code? Does it work fine without your additional changes?

Hi, I am still getting the above errors on the latest vLLM (i.e. v0.9).

[mgoin]

How are you triggering the above error? Just by serving llama 70b i.e. vllm serve meta-llama/Llama-3.3-70B-Instruct -tp 8? It would help to have a command or script to reproduce, as well as sharing the details of your environment

[rajagond]

Currently, I don't have access to an 8xH100 setup, but I was using the vllm/vllm-openai:v0.9.1 Docker image on an 8xH100 machine.

I added the following lines to vllm/v1/worker/gpu_model_runner.py inside the GPUModelRunner class.

self.query_first_loc = torch.zeros(self.max_num_reqs + 1,
                                   dtype=torch.int32,
                                   device=self.device)
self.query_second_loc = torch.zeros(self.max_num_reqs + 1,
                                    dtype=torch.int32,
                                    device=self.device)
self.seq_lens_first = torch.zeros(self.max_num_reqs,
                                  dtype=torch.int32,
                                  device=self.device)
self.seq_lens_second = torch.zeros(self.max_num_reqs,
                                   dtype=torch.int32,
                                   device=self.device)

I was running the following example:

import argparse
import dataclasses
import json
import time
from pathlib import Path
from typing import List, Optional
import random
import numpy as np
import torch
from tqdm import tqdm
import sys
from vllm import LLM, SamplingParams
from vllm.engine.arg_utils import EngineArgs
from vllm.inputs import PromptType
from vllm.utils import FlexibleArgumentParser
import os

# Seed setup for reproducibility
seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

# Sampling parameters
sampling_params = SamplingParams(
    n=1,
    temperature=1.0,
    top_p=0.8,
    seed=seed,
    max_tokens=100,
)
print(sampling_params)

# Environment settings
os.environ["VLLM_TORCH_PROFILER_DIR"] = "./logs"
os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_ATTENTION_BACKEND"] = "FLASH_ATTN_VLLM_V1"

# -------------------------
# Argument parser for model
# -------------------------
parser = argparse.ArgumentParser(description="Run inference with a selected model.")
parser.add_argument(
    "--model",
    choices=["mixtral", "qwen2", "llama3"],
    required=True,
    help="Select the model to use: mixtral, qwen2, or llama3",
)
args = parser.parse_args()

# Load model based on user choice
if args.model == "mixtral":
    llm = LLM(
        model="mistralai/Mixtral-8x22B-Instruct-v0.1",
        tokenizer_mode="mistral",
        tensor_parallel_size=8,
        enforce_eager=True,
        disable_custom_all_reduce=True,
        enable_chunked_prefill=True,
        max_num_batched_tokens=2048,
        enable_prefix_caching=False,
    )
elif args.model == "qwen2":
    llm = LLM(
        model="Qwen/Qwen2.5-72B-Instruct",
        tensor_parallel_size=8,
        enforce_eager=True,
        disable_custom_all_reduce=True,
        enable_chunked_prefill=True,
        max_num_batched_tokens=2048,
        enable_prefix_caching=False,
    )
else:  # llama3
    llm = LLM(
        model="meta-llama/Llama-3.3-70B-Instruct",
        tensor_parallel_size=8,
        enforce_eager=True,
        disable_custom_all_reduce=True,
        enable_chunked_prefill=True,
        max_num_batched_tokens=2048,
        enable_prefix_caching=False,
    )

prompts = [
    """
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments. The convergence of big data and neural networks has unlocked patterns previously invisible to human analysis, leading to breakthroughs in healthcare, climate science, and quantum computing. Yet, as we stand at this technological crossroads, questions arise about the nature of consciousness, creativity, and the ethical implications of AI development. Engineers and philosophers collaborate to ensure responsible innovation, while researchers push the boundaries of what's possible. Deep learning models process vast amounts of information, learning to recognize speech, generate art, and even engage in meaningful conversations. The integration of AI into society continues to accelerate, promising solutions to global challenges while raising important questions about privacy, accountability, and the future of human-machine collaboration. As we navigate this rapidly evolving landscape, the importance of maintaining human values and ethical considerations becomes increasingly crucial. The potential of artificial intelligence extends beyond mere computation, touching upon fundamental aspects of human existence and our collective future. This technological renaissance demands thoughtful discussion and careful consideration of its implications for generations to come.
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments.
    """,
    """
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments. The convergence of big data and neural networks has unlocked patterns previously invisible to human analysis, leading to breakthroughs in healthcare, climate science, and quantum computing. Yet, as we stand at this technological crossroads, questions arise about the nature of consciousness, creativity, and the ethical implications of AI development. Engineers and philosophers collaborate to ensure responsible innovation, while researchers push the boundaries of what's possible. Deep learning models process vast amounts of information, learning to recognize speech, generate art, and even engage in meaningful conversations. The integration of AI into society continues to accelerate, promising solutions to global challenges while raising important questions about privacy, accountability, and the future of human-machine collaboration. As we navigate this rapidly evolving landscape, the importance of maintaining human values and ethical considerations becomes increasingly crucial. The potential of artificial intelligence extends beyond mere computation, touching upon fundamental aspects of human existence and our collective future. This technological renaissance demands thoughtful discussion and careful consideration of its implications for generations to come.
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments.
    """,
    """
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments. The convergence of big data and neural networks has unlocked patterns previously invisible to human analysis, leading to breakthroughs in healthcare, climate science, and quantum computing. Yet, as we stand at this technological crossroads, questions arise about the nature of consciousness, creativity, and the ethical implications of AI development. Engineers and philosophers collaborate to ensure responsible innovation, while researchers push the boundaries of what's possible. Deep learning models process vast amounts of information, learning to recognize speech, generate art, and even engage in meaningful conversations. The integration of AI into society continues to accelerate, promising solutions to global challenges while raising important questions about privacy, accountability, and the future of human-machine collaboration. As we navigate this rapidly evolving landscape, the importance of maintaining human values and ethical considerations becomes increasingly crucial. The potential of artificial intelligence extends beyond mere computation, touching upon fundamental aspects of human existence and our collective future. This technological renaissance demands thoughtful discussion and careful consideration of its implications for generations to come.
    The dawn of the 21st century, artificial intelligence emerged as a transformative force, reshaping our understanding of human potential and technological progress. Machine learning algorithms, once confined to academic research papers, now power our daily interactions, from virtual assistants anticipating our needs to autonomous systems navigating complex environments.
    """,
]
outputs = llm.generate(prompts, sampling_params=sampling_params, use_tqdm=False)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt 1: \n Generated text: {generated_text!r}")

[Bluedyson]

The same issue occurred. I encapsulated vllm [0.9.1] and ray [0.27.1] myself and put them into a Docker container, but I used v0 instead of v1. However, an error is reported on the cloud provider's L20, but the same problem does not occur on the A100 and A800.

[Chris113113]

I'm seeing something similar, also on Llama3.3-70b. I am specifically running on GKE on 8xB200. I don't see this error when running on my local machine (also 8xB200).

I am running main/benchmarks/benchmark_throughput.py and see the below error.

I am using the vllm/vllm-openai:v0.9.1 image on Cuda 12.8.

(VllmWorker rank=0 pid=622) DEBUG 07-10 14:42:58 [parallel_state.py:918] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=5 pid=627) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=5 local_rank=5 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=6 pid=628) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=6 local_rank=6 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=1 pid=623) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=4 pid=626) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=4 local_rank=4 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=3 pid=625) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=7 pid=629) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=7 local_rank=7 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=2 pid=624) DEBUG 07-10 14:42:59 [parallel_state.py:918] world_size=8 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:51200 backend=nccl
(VllmWorker rank=1 pid=623) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=0 pid=622) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=6 pid=628) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=1 pid=623) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=2 pid=624) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=3 pid=625) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=7 pid=629) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=5 pid=627) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=6 pid=628) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=0 pid=622) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=4 pid=626) INFO 07-10 14:42:59 [utils.py:1126] Found nccl from library libnccl.so.2
(VllmWorker rank=2 pid=624) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=3 pid=625) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=7 pid=629) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=5 pid=627) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker rank=4 pid=626) INFO 07-10 14:42:59 [pynccl.py:70] vLLM is using nccl==2.26.2
DEBUG 07-10 14:43:02 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:43:12 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:43:22 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:43:32 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:43:42 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:43:52 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:02 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:12 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:22 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:32 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:42 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:44:52 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:02 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:12 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:22 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:32 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:42 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:45:52 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:02 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:12 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:22 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:32 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:43 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:46:53 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:03 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:13 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:23 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:33 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:43 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:47:53 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:03 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:13 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:23 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:33 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:43 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:48:53 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:49:03 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:49:13 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:49:23 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
DEBUG 07-10 14:49:33 [utils.py:485] Waiting for 1 local, 0 remote core engine proc(s) to start.
NCCL version 2.26.2+cuda12.2
ERROR 07-10 14:49:36 [core.py:515] EngineCore failed to start.
ERROR 07-10 14:49:36 [core.py:515] Traceback (most recent call last):
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
ERROR 07-10 14:49:36 [core.py:515]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 07-10 14:49:36 [core.py:515]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
ERROR 07-10 14:49:36 [core.py:515]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 76, in __init__
ERROR 07-10 14:49:36 [core.py:515]     self.model_executor = executor_class(vllm_config)
ERROR 07-10 14:49:36 [core.py:515]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 07-10 14:49:36 [core.py:515]     self._init_executor()
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 98, in _init_executor
ERROR 07-10 14:49:36 [core.py:515]     self.workers = WorkerProc.wait_for_ready(unready_workers)
ERROR 07-10 14:49:36 [core.py:515]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-10 14:49:36 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 427, in wait_for_ready
ERROR 07-10 14:49:36 [core.py:515]     raise e from None
ERROR 07-10 14:49:36 [core.py:515] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

--- STDERR ---
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[rank4]:[W710 14:49:34.202436919 TCPStore.cpp:125] [c10d] recvValue failed on SocketImpl(fd=199, addr=[localhost]:53342, remote=[localhost]:51200): failed to recv, got 0 bytes
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:678 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x98 (0x7f055b91e5e8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x5ba8afe (0x7f05b0a3cafe in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x5baae40 (0x7f05b0a3ee40 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: <unknown function> + 0x5bab74a (0x7f05b0a3f74a in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) + 0x2a9 (0x7f05b0a391a9 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::ProcessGroupNCCL::heartbeatMonitor() + 0x379 (0x7f055c699989 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xdc253 (0x7f054c9b3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #7: <unknown function> + 0x94ac3 (0x7f05cd6ddac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #8: <unknown function> + 0x126850 (0x7f05cd76f850 in /lib/x86_64-linux-gnu/libc.so.6)
[rank4]:[W710 14:49:34.208592459 ProcessGroupNCCL.cpp:1659] [PG ID 0 PG GUID 0 Rank 4] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: failed to recv, got 0 bytes
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 519, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 76, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 98, in _init_executor
    self.workers = WorkerProc.wait_for_ready(unready_workers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 427, in wait_for_ready
    raise e from None
Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Traceback (most recent call last):
  File "/home/vllm/vllm-sweep-benchmark/scripts/vllm-repo-benchmark/benchmarks/benchmark_vllm_throughput.py", line 517, in <module>
    main(args)
  File "/home/vllm/vllm-sweep-benchmark/scripts/vllm-repo-benchmark/benchmarks/benchmark_vllm_throughput.py", line 208, in main
    elapsed_time, request_outputs = run_vllm(
                                    ^^^^^^^^^
  File "/home/vllm/vllm-sweep-benchmark/scripts/vllm-repo-benchmark/benchmarks/benchmark_vllm_throughput.py", line 50, in run_vllm
    llm = LLM(**dataclasses.asdict(engine_args))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 243, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 501, in from_engine_args
    return engine_cls.from_vllm_config(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 124, in from_vllm_config
    return cls(vllm_config=vllm_config,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 101, in __init__
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 9 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
    self.engine_core = EngineCoreClient.make_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 75, in make_client
    return SyncMPClient(vllm_config, executor_class, log_stats)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 558, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 422, in __init__
    self._init_engines_direct(vllm_config, local_only,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 491, in _init_engines_direct
    self._wait_for_engine_startup(handshake_socket, input_address,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 511, in _wait_for_engine_startup
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 494, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_0': 1}

[gerayking]

I have same problem over vllm 0.9.2 for Qwen2.5-VL-32B on H20*4, tp4