[Bug]: Segmentation fault (core dumped) while loading deepseek coder v2 lite model

[zxdvd]

Your current environment

Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.1
Libc version: glibc-2.35

Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB
Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Address sizes:       46 bits physical, 57 bits virtual
Byte Order:          Little Endian
CPU(s):              112
On-line CPU(s) list: 0-111
Vendor ID:           GenuineIntel
Model name:          Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz
CPU family:          6
Model:               106
Thread(s) per core:  2
Core(s) per socket:  28
Socket(s):           2
Stepping:            6
BogoMIPS:            5187.80
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear spec_ctrl intel_stibp arch_capabilities
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           2.6 MiB (56 instances)
L1i cache:           1.8 MiB (56 instances)
L2 cache:            70 MiB (56 instances)
L3 cache:            96 MiB (2 instances)
NUMA node(s):        2
NUMA node0 CPU(s):   0-55
NUMA node1 CPU(s):   56-111

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] sentence-transformers==2.6.1
[pip3] torch==2.3.0
[pip3] torchvision==0.18.0
[pip3] transformers==4.42.3
[pip3] triton==2.3.0
[pip3] vllm_nccl_cu12==2.18.1.0.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.5.0.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity
GPU0     X      56-111  1

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug

The vllm is compiled from latest source code(commit af9ad46). It worked without problem for other model like opt-125m but always crashed with deepseek coder v2 lite.

And when trying to debug it with export VLLM_TRACE_FUNCTION=1, it didn't crash. After unset it, it crashed again.

zxd@zxd-cuda121-0:/code/code-complete$ export VLLM_TRACE_FUNCTION=1

zxd@zxd-cuda121-0:/code/code-complete$ python3.11 testdp2.py
INFO 07-01 05:20:47 llm_engine.py:169] Initializing an LLM engine (v0.5.0.post1) with config: model='/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123', speculative_config=None, tokenizer='/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
WARNING 07-01 05:20:47 logger.py:146] VLLM_TRACE_FUNCTION is enabled. It will record every function executed by Python. This will slow down the code. It is suggested to be used for debugging hang or crashes only.
INFO 07-01 05:20:47 logger.py:150] Trace frame log is saved to /tmp/vllm/vllm-instance-631f378da60646139e5c846da27af5d7/VLLM_TRACE_FUNCTION_for_process_32095_thread_139855533970048_at_2024-07-01_05:20:47.470343.log
DEBUG 07-01 05:20:49 parallel_state.py:788] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.18.162.132:39522 backend=nccl
Cache shape torch.Size([163840, 64])
INFO 07-01 05:21:15 model_runner.py:234] Loading model weights took 29.3010 GB
INFO 07-01 05:21:17 gpu_executor.py:83] # GPU blocks: 672, # CPU blocks: 606
Processed prompts: 100%|█| 2/2 [00:02<00:00,  1.29s/it, est. speed input: 5.44 toks/s, output:
Prompt: 'Hello, my name is', Generated text: ' ***\\<Your Name\\>*** and I am a ***\\<Your Profession'
Prompt: 'The president of the United States is', Generated text: ' not only the leader of the free world but also the commander-in-chief'


zxd@zxd-cuda121-0:/code/code-complete$ unset VLLM_TRACE_FUNCTION

zxd@zxd-cuda121-0:/code/code-complete$ python3.11 testdp2.py
INFO 07-01 05:23:43 llm_engine.py:169] Initializing an LLM engine (v0.5.0.post1) with config: model='/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123', speculative_config=None, tokenizer='/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None), seed=0, served_model_name=/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct-24-06-17-1123)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
DEBUG 07-01 05:23:45 parallel_state.py:788] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://10.18.162.132:38231 backend=nccl
Cache shape torch.Size([163840, 64])
INFO 07-01 05:24:07 model_runner.py:234] Loading model weights took 29.3010 GB
Segmentation fault (core dumped)

I'm trying to debug the core with gdb python <THE_CORE_FILE> but didn't find something useful. Anyone can help me about how to get more information from the core file?

Following is code to reproduce.

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

model_name = '/data/models/deepseek/deepseek-ai__deepseek-coder-v2-lite-instruct'
# model_name = '/data/models/opt-125m'
llm = LLM(model=model_name, trust_remote_code=True, max_model_len=8192, enforce_eager=True)

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

[youkaichao]

And when trying to debug it with export VLLM_TRACE_FUNCTION=1, it didn't crash. After unset it, it crashed again.

that's very strange ...

Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] (64-bit runtime)

One thing I notice is you are using a release candidate version of Python. Do you try to switch between several python versions? e.g. official release of Python 3.10 / 3.11 ?

[zxdvd]

And when trying to debug it with export VLLM_TRACE_FUNCTION=1, it didn't crash. After unset it, it crashed again.

that's very strange ...

Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] (64-bit runtime)

One thing I notice is you are using a release candidate version of Python. Do you try to switch between several python versions? e.g. official release of Python 3.10 / 3.11 ?

OK. I'll try with python 3.11 release. The current 3.11 rc1 is from image nvidia/cuda:12.1.0-devel-ubuntu22.04.

[zxdvd]

I switched to python 3.10 and didn't reproduce crash.

[fengyang95]

I encountered this issue in Python 3.9 as well.

[fengyang95]

From my testing, it appears to be random; sometimes it core dumps, and sometimes it doesn't.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!

[anton164]

I encountered a similar segmentation fault issue as described in this issue with the later vLLM versions (0.5.4+) on Databricks Runtime 15.x. The seg fault happens consistently when using enable_lora=True and does not occur when setting VLLM_TRACE_FUNCTION=1.

Upon closer inspection it seems like this issue is caused by this async issue in cpython which was fixed in cPython 3.11.8.

The seg fault issue does not occur when running vLLM + 3.11.10. I believe vLLM might be generally broken on any earlier version of Python 3.11 (pre 3.11.8)

[vatsal-kr]

I still encounter this issue in vllm=0.6.4post1 with python 3.12 and python 3.11.10
Anyone managed to find a fix?