Skip to content

Conversation

hjh0119
Copy link
Collaborator

@hjh0119 hjh0119 commented May 21, 2025

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fix the issue where only one choice is returned when using V1 Engine and n > 1

vllm-project/vllm#7381

The same logic can be found in vllm.LLM:
https://github.com/vllm-project/vllm/blob/v0.8.5/vllm/entrypoints/llm.py#L1347-L1350

The response aggregation logic in the VLLM V1 Engine is detailed here:
https://github.com/vllm-project/vllm/blob/v0.8.5/vllm/v1/engine/parallel_sampling.py#L101-L110

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

@hjh0119
Copy link
Collaborator Author

hjh0119 commented May 21, 2025

test code (VLLM_USE_V1 = 1)

    from swift.llm import InferEngine, InferRequest, RequestConfig
    from swift.plugin import InferStats
    model = 'Qwen/Qwen2.5-1.5B-Instruct'
    from swift.llm import VllmEngine
    engine = VllmEngine(model, gpu_memory_utilization=0.6, use_async_engine=False, max_model_len=8192)
    req = InferRequest(messages=[{'role': 'user', 'content': 'Hello!'}])
    output = engine.infer([req], RequestConfig(n=4))

@hjh0119 hjh0119 merged commit d23e717 into modelscope:main May 21, 2025
1 of 2 checks passed
@hjh0119 hjh0119 deleted the v1n branch May 21, 2025 07:08
tastelikefeet added a commit to tastelikefeet/swift that referenced this pull request May 23, 2025
…o_padding_ulysses

* commit 'e9475f1a306614b30fc6314cc08eb5b40a3f17aa':
  qwen2_5_vl support video use image_dir (modelscope#4326)
  [megatron] Add extra args and provider support for easily customize megatron (modelscope#4240)
  Update internvl.py, solve the exception when setting customized INPUT_SIZE. (modelscope#4320)
  [grpo] support liger loss (modelscope#3781)
  compat transformer_engine update (modelscope#4317)
  compat transformers==4.52 (modelscope#4308)
  [grpo] support dp in external mode (modelscope#4279)
  fix vllm engine return empty in stream generation (modelscope#4303)
  fix (modelscope#4316)
  update swift image (modelscope#4309)
  update load_args (modelscope#4296)
  fix n > 1 with vLLM V1 Engine (modelscope#4295)
  Reuse existing code
  [grpo] fix num of reward_model > 1  (modelscope#4287)
  modify grpo system
  fix grpo tab
  support grpo web_ui

# Conflicts:
#	swift/trainers/sequence_parallel/ulysses.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants