[Bug]: deepseek-r1 on A800

[shilijunConnan]

Your current environment

8 *a800 * 8
vllm 0.7.1
ray

🐛 Describe the bug

I used Ray and 8 8*A800s to run deepseek-r1, set tp=8, pp=8, gpu utilization was 0.98, but one machine's video memory usage was very low, while the other machine's usage was close to the limit.

Is there any way to balance the two machines?
In the follow picture， machine 1 and 3：

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[founture]

Does the DeepSeek-R1 671B @have correct output results on A800? #12954

[jyyang621]

A800不是不支持FP8吗，难道是用的bf16版本？

[Bow-Lin]

Could you please help paste the inference and model loading part codes?

[ww0o0]

I encountered an issue while using Ray to deploy VLLM on 64 NVIDIA GeForce RTX 4090 GPUs. The problem manifests as follows: among multiple servers, one server consumes significantly more VRAM than others. Additionally, when running the VLLM command, the --enforce-eager parameter must be used; otherwise, an error occurs. In theory, on a setup with 64 RTX 4090 GPUs, the system should run smoothly even without the --enforce-eager parameter.

Here is the error message when the --enforce-eager parameter is not used:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 246.00 MiB. GPU 0 has a total capacity of 23.65 GiB of which 178.06 MiB is free. Process 60555 has 23.46 GiB memory in use. Of the allocated memory 21.10 GiB is allocated by PyTorch, with 82.00 MiB allocated in private pools (e.g., CUDA Graphs), and 98.34 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

After adding the --enforce-eager parameter, the program runs successfully, but it results in uneven VRAM usage across servers, with some servers consuming significantly more VRAM than others.

Here is the command used to run VLLM:
vllm serve /home/models/ --served_model_name deepseek_r1 --tensor-parallel-size 8 --pipeline-parallel-size 8 --gpu-memory-utilization 0.95 --trust-remote-code --enforce-eager

[e1ijah1]

I've encountered the same issue #12895 . This appears to be caused by uneven pipeline parallel partitioning in vLLM. Moreover, since the last node in the Ray cluster has the least available GPU memory after service initialization, the vLLM service is prone to failure when this last node runs out of memory.

[yangxin120704]

I encountered an issue while using Ray to deploy VLLM on 64 NVIDIA GeForce RTX 4090 GPUs. The problem manifests as follows: among multiple servers, one server consumes significantly more VRAM than others. Additionally, when running the VLLM command, the --enforce-eager parameter must be used; otherwise, an error occurs. In theory, on a setup with 64 RTX 4090 GPUs, the system should run smoothly even without the --enforce-eager parameter.

Here is the error message when the --enforce-eager parameter is not used: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 246.00 MiB. GPU 0 has a total capacity of 23.65 GiB of which 178.06 MiB is free. Process 60555 has 23.46 GiB memory in use. Of the allocated memory 21.10 GiB is allocated by PyTorch, with 82.00 MiB allocated in private pools (e.g., CUDA Graphs), and 98.34 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

After adding the --enforce-eager parameter, the program runs successfully, but it results in uneven VRAM usage across servers, with some servers consuming significantly more VRAM than others.

Here is the command used to run VLLM: vllm serve /home/models/ --served_model_name deepseek_r1 --tensor-parallel-size 8 --pipeline-parallel-size 8 --gpu-memory-utilization 0.95 --trust-remote-code --enforce-eager

What is the TPS performance with 64 RTX 4090 GPUs, and what is the network card bandwidth in Gbps?

[hmellor]

It is expected that the first and last rank will have higher memory usage because:

• The first rank contains the input embeddings
• The last rank contains the output embeddings

However, I have identified a bug which meant that the output embeddings were present on all ranks. This meant that the first rank was using more memory than necessary.

This bug should be fixed by #13833.

[hmellor]

The other half of this issue is that DeepSeek R1 has 61 hidden layers. Currently, if the number of hidden layers is not divisible by the pipeline world size, the extra layers go on the last rank.

This means that the hidden layers are distributed as follows:

0. 7
1. 7
2. 7
3. 7
4. 7
5. 7
6. 7
7. 7 + 5 = 12

A more sensible strategy would be to distribute the extra layers across the ranks between the first and last ranks.

[hmellor]

I'll try and make a PR that handles this automatically, but in the meantime could you try setting VLLM_PP_LAYER_PARTITION=7,8,8,8,8,8,7,7?