[Bug]: BGE Embedder not working with V1 Engine #22892
Description
Your current environment
The output of python collect_env.py
Collecting environment information...
uv is set
==============================
System Info
==============================
OS : Rocky Linux 9.6 (Blue Onyx) (x86_64)
GCC version : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Clang version : Could not collect
CMake version : Could not collect
Libc version : glibc-2.34
==============================
PyTorch Info
==============================
PyTorch version : 2.9.0.dev20250804+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.5 (main, Aug 14 2024, 05:08:31) [Clang 18.1.8 ] (64-bit runtime)
Python platform : Linux-6.8.0-64-generic-x86_64-with-glibc2.34
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to :
GPU models and configuration : GPU 0: NVIDIA L40S
Nvidia driver version : 550.127.08
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7413 24-Core Processor
CPU family: 25
Model: 1
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 1
Stepping: 1
BogoMIPS: 5300.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid fsrm arch_capabilities
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 512 KiB (8 instances)
L1i cache: 512 KiB (8 instances)
L2 cache: 4 MiB (8 instances)
L3 cache: 128 MiB (8 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.13.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.3.9
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] pynvml==12.0.0
[pip3] pytorch-triton==3.4.0+git11ec6354
[pip3] pyzmq==27.0.1
[pip3] torch==2.9.0.dev20250804+cu128
[pip3] torchaudio==2.8.0.dev20250804+cu128
[pip3] torchvision==0.24.0.dev20250804+cu128
[pip3] transformers==4.55.0
[pip3] triton==3.4.0+git663e04e8
[pip3] triton-kernels==1.0.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.10.2.dev2+gf5635d62e.d20250807 (git sha: f5635d62e, date: 20250807)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-7 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/lib64
NCCL_VERSION=2.25.1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NVIDIA_VISIBLE_DEVICES=all
VLLM_CONFIGURE_LOGGING=1
VLLM_USE_V1=1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
🐛 Describe the bug
Hey guys,
I recently upgrade from V0 engine to V1 engine for my embedding server but I can't embed with V1 Engine for bge (bert based) model...
Here is the error logging of it:
INFO 08-14 08:44:47 [__init__.py:241] Automatically detected platform cuda.
WARNING 08-14 08:44:53 [__init__.py:1683] argument 'task' is deprecated
(APIServer pid=43) INFO 08-14 08:44:53 [api_server.py:1787] vLLM API server version 0.10.2.dev2+gf5635d62e.d20250807
(APIServer pid=43) INFO 08-14 08:44:53 [utils.py:326] non-default args: {'model_tag': 'BAAI/bge-large-en-v1.5', 'host': '0.0.0.0', 'port': 30000, 'model': 'BAAI/bge-large-en-v1.5', 'task': 'embed', 'dtype': 'float32', 'max_model_len': 512, 'served_model_name': ['bge-embedding'], 'hf_token': '***', 'override_pooler_config': {'pooling_type': 'MEAN', 'normalize': True}, 'gpu_memory_utilization': 0.125, 'max_num_seqs': 32}
(APIServer pid=43) INFO 08-14 08:44:54 [config.py:663] Found sentence-transformers tokenize configuration.
(APIServer pid=43) INFO 08-14 08:45:01 [config.py:726] Resolved architecture: BertModel
(APIServer pid=43) INFO 08-14 08:45:01 [config.py:561] Found sentence-transformers modules configuration.
(APIServer pid=43) INFO 08-14 08:45:01 [config.py:581] Found pooling configuration.
(APIServer pid=43) INFO 08-14 08:45:01 [config.py:1759] Using max model len 512
(APIServer pid=43) INFO 08-14 08:45:01 [arg_utils.py:1609] (Disabling) chunked prefill by default
(APIServer pid=43) INFO 08-14 08:45:01 [arg_utils.py:1612] (Disabling) prefix caching by default
(APIServer pid=43) INFO 08-14 08:45:02 [config.py:4821] Only "last" pooling supports chunked prefill and prefix caching; disabling both.
INFO 08-14 08:45:07 [__init__.py:241] Automatically detected platform cuda.
(EngineCore_0 pid=86) INFO 08-14 08:45:10 [core.py:654] Waiting for init message from front-end.
(EngineCore_0 pid=86) INFO 08-14 08:45:10 [core.py:73] Initializing a V1 LLM engine (v0.10.2.dev2+gf5635d62e.d20250807) with config: model='BAAI/bge-large-en-v1.5', speculative_config=None, tokenizer='BAAI/bge-large-en-v1.5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float32, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=bge-embedding, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, pooler_config=PoolerConfig(pooling_type='MEAN', normalize=True, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":64,"local_cache_dir":null}
(EngineCore_0 pid=86) W0814 08:45:10.919000 86 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
(EngineCore_0 pid=86) W0814 08:45:10.919000 86 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[W814 08:45:12.611949827 ProcessGroupNCCL.cpp:915] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_0 pid=86)
(EngineCore_0 pid=86) LL LL MMM MMM
(EngineCore_0 pid=86) LL LL MMMM MMMM
(EngineCore_0 pid=86) V LL LL MM MM MM MM
(EngineCore_0 pid=86) vvvv VVVV LL LL MM MM MM MM
(EngineCore_0 pid=86) vvvv VVVV LL LL MM MMM MM
(EngineCore_0 pid=86) vvv VVVV LL LL MM M MM
(EngineCore_0 pid=86) vvVVVV LL LL MM MM
(EngineCore_0 pid=86) VVVV LLLLLLLLLL LLLLLLLLL M M
(EngineCore_0 pid=86)
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [parallel_state.py:1102] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [gpu_model_runner.py:1913] Starting to load model BAAI/bge-large-en-v1.5...
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [gpu_model_runner.py:1945] Loading model from scratch...
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [cuda.py:340] Using FlexAttention backend for dtype=torch.float32 on V1 engine.
(EngineCore_0 pid=86) INFO 08-14 08:45:12 [weight_utils.py:296] Using model weights format ['*.safetensors']
(EngineCore_0 pid=86) INFO 08-14 08:45:13 [weight_utils.py:349] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 57.90it/s]
(EngineCore_0 pid=86)
(EngineCore_0 pid=86) INFO 08-14 08:45:13 [default_loader.py:262] Loading weights took 0.28 seconds
(EngineCore_0 pid=86) INFO 08-14 08:45:13 [gpu_model_runner.py:1962] Model loading took 1.2457 GiB and 1.150166 seconds
(EngineCore_0 pid=86) INFO 08-14 08:45:16 [backends.py:530] Using cache directory: /root/.cache/vllm/torch_compile_cache/557956057e/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_0 pid=86) INFO 08-14 08:45:16 [backends.py:541] Dynamo bytecode transform time: 2.36 s
Capturing CUDA graph shapes: 0%| | 0/11 [00:00<?, ?it/s]/app/.venv/lib/python3.12/site-packages/torch/backends/cuda/__init__.py:131: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:80.)
(EngineCore_0 pid=86) return torch._C._get_cublas_allow_tf32()
(EngineCore_0 pid=86) /app/.venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:305: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
(EngineCore_0 pid=86) warnings.warn(
(EngineCore_0 pid=86) INFO 08-14 08:45:18 [backends.py:194] Cache the graph for dynamic shape for later use
(EngineCore_0 pid=86) INFO 08-14 08:45:21 [backends.py:215] Compiling a graph for dynamic shape takes 4.82 s
(EngineCore_0 pid=86) INFO 08-14 08:45:26 [monitor.py:34] torch.compile takes 7.18 s in total
Capturing CUDA graph shapes: 100%|██████████| 11/11 [00:12<00:00, 1.18s/it]
(EngineCore_0 pid=86) INFO 08-14 08:45:27 [gpu_model_runner.py:2567] Graph capturing finished in 13 secs, took 0.12 GiB
(EngineCore_0 pid=86) INFO 08-14 08:45:27 [core.py:216] init engine (profile, create kv cache, warmup model) took 13.33 seconds
(EngineCore_0 pid=86) INFO 08-14 08:45:27 [config.py:663] Found sentence-transformers tokenize configuration.
(EngineCore_0 pid=86) INFO 08-14 08:45:27 [core.py:120] Disabling chunked prefill for model without KVCache
(EngineCore_0 pid=86) INFO 08-14 08:45:27 [config.py:4821] Only "last" pooling supports chunked prefill and prefix caching; disabling both.
(APIServer pid=43) INFO 08-14 08:45:27 [loggers.py:142] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 1
(APIServer pid=43) INFO 08-14 08:45:27 [api_server.py:1599] Supported_tasks: ['embed', 'encode']
(APIServer pid=43) INFO 08-14 08:45:27 [api_server.py:1857] Starting vLLM API server 0 on http://0.0.0.0:30000
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:29] Available routes are:
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /docs, Methods: HEAD, GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /redoc, Methods: HEAD, GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /health, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /load, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /ping, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /ping, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /tokenize, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /detokenize, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/models, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /version, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/responses, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/completions, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/embeddings, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /pooling, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /classify, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /score, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/score, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/audio/translations, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /rerank, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v1/rerank, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /v2/rerank, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /invocations, Methods: POST
(APIServer pid=43) INFO 08-14 08:45:27 [launcher.py:37] Route: /metrics, Methods: GET
(APIServer pid=43) INFO: Started server process [43]
(APIServer pid=43) INFO: Waiting for application startup.
(APIServer pid=43) INFO: Application startup complete.
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.10.2.dev2+gf5635d62e.d20250807) with config: model='BAAI/bge-large-en-v1.5', speculative_config=None, tokenizer='BAAI/bge-large-en-v1.5', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float32, max_seq_len=512, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=bge-embedding, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=False, pooler_config=PoolerConfig(pooling_type='MEAN', normalize=True, softmax=None, step_tag_id=None, returned_token_ids=None), compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.cache/vllm/torch_compile_cache/557956057e","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":64,"local_cache_dir":"/root/.cache/vllm/torch_compile_cache/557956057e/rank_0_0/backbone"},
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=embd-21c6997ecb014437bd4e0fc680bc9134-0,prompt_token_ids_len=4,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=None,block_ids=(),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={embd-21c6997ecb014437bd4e0fc680bc9134-0: 4}, total_num_scheduled_tokens=4, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=1.0, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] EngineCore encountered a fatal error.
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] Traceback (most recent call last):
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 711, in run_engine_core
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] engine_core.run_busy_loop()
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] self._process_engine_step()
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 763, in _process_engine_step
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] outputs, model_executed = self.step_fn()
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 302, in step
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] model_output = self.execute_model_with_error_logging(
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 276, in execute_model_with_error_logging
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] raise err
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 267, in execute_model_with_error_logging
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return model_fn(scheduler_output)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] output = self.collective_rpc("execute_model",
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 58, in collective_rpc
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] answer = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/utils/__init__.py", line 2948, in run_method
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return func(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return func(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 362, in execute_model
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] output = self.model_runner.execute_model(scheduler_output,
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) Process EngineCore_0:
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return func(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1568, in execute_model
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] model_output = self.model(
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return forward_call(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/model_executor/models/bert.py", line 475, in forward
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self.model(input_ids=input_ids,
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 279, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] model_output = self.forward(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/model_executor/models/bert.py", line 358, in forward
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] def forward(
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 413, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return super().__call__(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return forward_call(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1005, in _fn
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return fn(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 837, in call_wrapped
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._wrapped_call(self, *args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 413, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] raise e
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return forward_call(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "<eval_with_key>.50", line 306, in forward
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] submod_1 = self.submod_1(getitem, s72, getitem_1, getitem_2, getitem_3); getitem = getitem_1 = getitem_2 = submod_1 = None
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 837, in call_wrapped
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._wrapped_call(self, *args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 413, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] raise e
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/fx/graph_module.py", line 400, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc]
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._call_impl(*args, **kwargs)
(EngineCore_0 pid=86) Traceback (most recent call last):
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return forward_call(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "<eval_with_key>.2", line 5, in forward
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] unified_attention_with_output = torch.ops.vllm.unified_attention_with_output(query, key, value, output_1, 'model.encoder.layer.0.attention.output.attn'); query = key = value = output_1 = unified_attention_with_output = None
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/torch/_ops.py", line 1254, in __call__
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] return self._op(*args, **kwargs)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/attention/layer.py", line 488, in unified_attention_with_output
(EngineCore_0 pid=86) File "/root/.local/share/uv/python/cpython-3.12.5-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] self.impl.forward(self,
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/attention/backends/flex_attention.py", line 428, in forward
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] key_cache, value_cache = kv_cache.unbind(0)
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_0 pid=86) ERROR 08-14 08:45:48 [core.py:720] ValueError: not enough values to unpack (expected 2, got 0)
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] AsyncLLM output_handler failed.
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] Traceback (most recent call last):
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 389, in output_handler
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] outputs = await engine_core.get_output_async()
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] File "/app/.venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 809, in get_output_async
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] raise self._format_exception(outputs) from None
(APIServer pid=43) ERROR 08-14 08:45:48 [async_llm.py:430] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.