[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue #15906

njhill · 2025-04-01T20:59:12Z

So that a single queue / socket address can be used for all engines.

This also flips the front-end to bind the input socket and engines to connect to it, and changes to use this socket for the startup ready signal rather than a separate pipe which avoids a race condition (where the front-end could start sending messages before the engine has finished connecting) and will also be needed for remote engines.

This is in preparation for multi-node support.

github-actions · 2025-04-01T20:59:21Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

… queue So that a single queue / socket address can be used for all engines. Signed-off-by: Nick Hill <[email protected]>

Signed-off-by: Nick Hill <[email protected]>

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: xinyuxiao <[email protected]>

hongxiayang · 2025-04-05T23:28:53Z

@njhill Does this work on AMD GPU as well?

hongxiayang · 2025-04-05T23:37:26Z

vllm/v1/engine/core_client.py

+        identities = set(eng.index for eng in self.resources.core_engines)
+        while identities:
+            while not sync_input_socket.poll(timeout=STARTUP_POLL_PERIOD_MS):
+                logger.info("Waiting for %d core engine proc(s) to start: %s",


what happens this run forever and never successful?

@hongxiayang good point... the design here is for accommodating remote engines which might be started at different times, but we should still exit when one of the local engines fails. I'll make an update to address this.

@hongxiayang FYI here's the fix #16137

njhill · 2025-04-06T00:34:03Z

@njhill Does this work on AMD GPU as well?

@hongxiayang it should be back-end agnostic, it doesn't touch that layer.

Could you give more detail about how it broke AMD GPU case?

…put queue (#15906)" This reverts commit 651cf0f.

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>

…put queue (vllm-project#15906)" This reverts commit 651cf0f.

…s for input queue (vllm-project#15906)"" This reverts commit 296c657.

@LucasWilkinson

SUMMARY: sync to upstream `v0.8.4` and cherry-pick of `7eb42556281d30436a3a988f2c9184ec63c59338`. the cherry-pick is @LucasWilkinson 's llama4 patch. GIT LOG: ```bash commit b197179 (HEAD -> sync-upstream-v0.8.4, origin/sync-upstream-v0.8.4) Author: Lucas Wilkinson <[email protected]> Date: Fri Apr 18 01:13:29 2025 -0400 [BugFix] Accuracy fix for llama4 int4 - improperly casted scales (vllm-project#16801) Signed-off-by: Lucas Wilkinson <[email protected]> commit 60267cc Author: andy-neuma <[email protected]> Date: Mon Apr 21 14:57:52 2025 -0400 remove duplicate entries commit 9d18b50 Merge: db0e117 dc1b4a6 Author: andy-neuma <[email protected]> Date: Mon Apr 21 14:50:01 2025 -0400 Merge remote-tracking branch 'upstream/v0.8.4' into sync-upstream-v0.8.4 commit db0e117 Author: andy-neuma <[email protected]> Date: Mon Apr 21 14:35:23 2025 -0400 Revert "Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (vllm-project#15906)"" This reverts commit 296c657. commit dc1b4a6 (tag: v0.8.4, upstream/v0.8.4) Author: Russell Bryant <[email protected]> Date: Sun Apr 13 22:13:38 2025 -0400 [Core][V0] Enable regex support with xgrammar (vllm-project#13228) Signed-off-by: Russell Bryant <[email protected]> ``` COMMANDS: ```bash git fetch upstream git checkout -b sync-upstream-v0.8.4 git revert 296c657 git merge upstream/v0.8.4 git cherry-pick 7eb4255 ``` TEST PLAN: accept sync ... https://github.com/neuralmagic/nm-cicd/actions/runs/14581880024 release ... https://github.com/neuralmagic/nm-cicd/actions/runs/14596026989 --------- Signed-off-by: Tristan Leclercq <[email protected]> Signed-off-by: yihong0618 <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: chaunceyjiang <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]> Signed-off-by: Jonghyun Choe <[email protected]> Signed-off-by: Lu Fang <[email protected]> Signed-off-by: Hyesoo Yang <[email protected]> Signed-off-by: Ben Jackson <[email protected]> Signed-off-by: Roger Wang <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: rongfu.leng <[email protected]> Signed-off-by: Varun Sundar Rabindranath <[email protected]> Signed-off-by: paolovic <[email protected]> Signed-off-by: Chengji Yao <[email protected]> Signed-off-by: Kay Yan <[email protected]> Signed-off-by: Woosuk Kwon <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: shen-shanshan <[email protected]> Signed-off-by: YamPengLi <[email protected]> Signed-off-by: WangErXiao <[email protected]> Signed-off-by: Aston Zhang <[email protected]> Signed-off-by: Chris Thi <[email protected]> Signed-off-by: drisspg <[email protected]> Signed-off-by: Jon Swenson <[email protected]> Signed-off-by: Keyun Tong <[email protected]> Signed-off-by: Lu Fang <[email protected]> Signed-off-by: Xiaodong Wang <[email protected]> Signed-off-by: Yang Chen <[email protected]> Signed-off-by: Ye (Charlotte) Qi <[email protected]> Signed-off-by: Yong Hoon Shin <[email protected]> Signed-off-by: Zijing Liu <[email protected]> Signed-off-by: Lu Fang <[email protected]> Signed-off-by: Lucia Fang <[email protected]> Signed-off-by: Gregory Shtrasberg <[email protected]> Signed-off-by: NickLucche <[email protected]> Signed-off-by: Benjamin Chislett <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Leon Seidel <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: youkaichao <[email protected]> Signed-off-by: Miles Williams <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Siyuan Liu <[email protected]> Signed-off-by: Kebe <[email protected]> Signed-off-by: simon-mo <[email protected]> Signed-off-by: Alex-Brooks <[email protected]> Signed-off-by: Tianyuan Wu <[email protected]> Signed-off-by: imkero <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Russell Bryant <[email protected]> Signed-off-by: Jee Jee Li <[email protected]> Signed-off-by: Yue <[email protected]> Signed-off-by: tjtanaa <[email protected]> Signed-off-by: kliuae <[email protected]> Signed-off-by: luka <[email protected]> Signed-off-by: lvfei.lv <[email protected]> Signed-off-by: Ajay Vohra <[email protected]> Signed-off-by: Guillaume Calmettes <[email protected]> Signed-off-by: zh Wang <[email protected]> Signed-off-by: Chendi Xue <[email protected]> Signed-off-by: Joe Runde <[email protected]> Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: Aaron Ang <[email protected]> Signed-off-by: Benjamin Kitor <[email protected]> Signed-off-by: Michael Goin <[email protected]> Signed-off-by: Chenyaaang <[email protected]> Signed-off-by: cyy <[email protected]> Signed-off-by: wineandchord <[email protected]> Signed-off-by: LiuXiaoxuanPKU <[email protected]> Signed-off-by: Chih-Chieh-Yang <[email protected]> Signed-off-by: look <[email protected]> Signed-off-by: jadewang21 <[email protected]> Signed-off-by: alexey-belyakov <[email protected]> Signed-off-by: jiang.li <[email protected]> Signed-off-by: DefTruth <[email protected]> Signed-off-by: chaow <[email protected]> Signed-off-by: Tomasz Zielinski <[email protected]> Signed-off-by: rzou <[email protected]> Signed-off-by: Travis Johnson <[email protected]> Signed-off-by: Christian Sears <[email protected]> Signed-off-by: Gogs <[email protected]> Signed-off-by: Yuan Tang <[email protected]> Signed-off-by: Tianer Zhou <[email protected]> Signed-off-by: [email protected] <[email protected]> Signed-off-by: Jie Fu <[email protected]> Signed-off-by: snowcharm <[email protected]> Signed-off-by: Ryan McConville <[email protected]> Co-authored-by: Tristan Leclercq <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> Co-authored-by: yihong <[email protected]> Co-authored-by: Reid <[email protected]> Co-authored-by: reidliu41 <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: Chauncey <[email protected]> Co-authored-by: Jinzhen Lin <[email protected]> Co-authored-by: Jonghyun Choe <[email protected]> Co-authored-by: Lucia Fang <[email protected]> Co-authored-by: Hyesoo Yang <[email protected]> Co-authored-by: Ben Jackson <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Paul Schweigert <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: rongfu.leng <[email protected]> Co-authored-by: Varun Sundar Rabindranath <[email protected]> Co-authored-by: Varun Sundar Rabindranath <[email protected]> Co-authored-by: paolovic <[email protected]> Co-authored-by: paolovic <[email protected]> Co-authored-by: Chengji Yao <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> Co-authored-by: Martin Hoyer <[email protected]> Co-authored-by: Kay Yan <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Shanshan Shen <[email protected]> Co-authored-by: YamPengLi <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Co-authored-by: Robin <[email protected]> Co-authored-by: Lu Fang <[email protected]> Co-authored-by: Lu Fang <[email protected]> Co-authored-by: Roger Wang <[email protected]> Co-authored-by: Gregory Shtrasberg <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Benjamin Chislett <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: leon-seidel <[email protected]> Co-authored-by: Driss Guessous <[email protected]> Co-authored-by: Michael Goin <[email protected]> Co-authored-by: youkaichao <[email protected]> Co-authored-by: Miles Williams <[email protected]> Co-authored-by: Satyajith Chilappagari <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: Jennifer Zhao <[email protected]> Co-authored-by: zxfan-cpu <[email protected]> Co-authored-by: Yong Hoon Shin <[email protected]> Co-authored-by: Siyuan Liu <[email protected]> Co-authored-by: Kebe <[email protected]> Co-authored-by: Simon Mo <[email protected]> Co-authored-by: Alex Brooks <[email protected]> Co-authored-by: TY-AMD <[email protected]> Co-authored-by: wang.yuqi <[email protected]> Co-authored-by: Kero Liang <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Russell Bryant <[email protected]> Co-authored-by: Jee Jee Li <[email protected]> Co-authored-by: yueshen2016 <[email protected]> Co-authored-by: TJian <[email protected]> Co-authored-by: Hongxia Yang <[email protected]> Co-authored-by: kliuae <[email protected]> Co-authored-by: Luka Govedič <[email protected]> Co-authored-by: Accelerator1996 <[email protected]> Co-authored-by: ajayvohra2005 <[email protected]> Co-authored-by: Guillaume Calmettes <[email protected]> Co-authored-by: zh Wang <[email protected]> Co-authored-by: Chendi.Xue <[email protected]> Co-authored-by: Joe Runde <[email protected]> Co-authored-by: Yuxuan Zhang <[email protected]> Co-authored-by: Aaron Ang <[email protected]> Co-authored-by: Jintao <[email protected]> Co-authored-by: Benjamin Kitor <[email protected]> Co-authored-by: Chenyaaang <[email protected]> Co-authored-by: cyyever <[email protected]> Co-authored-by: Ye (Charlotte) Qi <[email protected]> Co-authored-by: wineandchord <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Lily Liu <[email protected]> Co-authored-by: Chih-Chieh Yang <[email protected]> Co-authored-by: Yu Chin Fabian Lim <[email protected]> Co-authored-by: look <[email protected]> Co-authored-by: WWW <[email protected]> Co-authored-by: Alexey Belyakov <[email protected]> Co-authored-by: Li, Jiang <[email protected]> Co-authored-by: DefTruth <[email protected]> Co-authored-by: chaow-amd <[email protected]> Co-authored-by: Tomasz Zielinski <[email protected]> Co-authored-by: Richard Zou <[email protected]> Co-authored-by: Travis Johnson <[email protected]> Co-authored-by: Kai Wu <[email protected]> Co-authored-by: Christian Sears <[email protected]> Co-authored-by: Gogs <[email protected]> Co-authored-by: Yuan Tang <[email protected]> Co-authored-by: Tianer Zhou <[email protected]> Co-authored-by: Huazhong Ji <[email protected]> Co-authored-by: Jie Fu (傅杰) <[email protected]> Co-authored-by: SnowCharm <[email protected]> Co-authored-by: Ryan McConville <[email protected]> Co-authored-by: andy-neuma <[email protected]>

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]>

…nput queue (vllm-project#15906)" This reverts commit 296c657.

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]>

…put queue (vllm-project#15906)" This reverts commit 651cf0f.

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Mu Huai <[email protected]>

njhill requested review from WoosukKwon, robertgshaw2-redhat, ywang96, comaniac and alexm-redhat as code owners April 1, 2025 20:59

mergify bot added the v1 label Apr 1, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER socket types for input…

41888d4

… queue So that a single queue / socket address can be used for all engines. Signed-off-by: Nick Hill <[email protected]>

njhill force-pushed the zmq-router branch from b75080f to 41888d4 Compare April 1, 2025 21:26

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 2, 2025

njhill added 4 commits April 2, 2025 09:54

Use input queue for engine ready signal

034bf9d

Signed-off-by: Nick Hill <[email protected]>

Fix async startup case

4f08c78

Signed-off-by: Nick Hill <[email protected]>

Fix offline DP compatibility

5fb0cf1

Signed-off-by: Nick Hill <[email protected]>

Merge remote-tracking branch 'refs/remotes/origin/main' into zmq-router

0d93d8c

njhill mentioned this pull request Apr 3, 2025

[V1] DP scale-out (2/N): Decouple engine process management and comms #15977

Merged

4 tasks

tlrmchlsmth mentioned this pull request Apr 3, 2025

[RFC]: Data Parallel Attention and Expert Parallel MoEs #16037

Open

37 tasks

simon-mo approved these changes Apr 4, 2025

View reviewed changes

simon-mo merged commit 651cf0f into vllm-project:main Apr 4, 2025
32 checks passed

njhill deleted the zmq-router branch April 4, 2025 20:03

Alex4210987 pushed a commit to LeiWang1999/vllm-bitblas that referenced this pull request Apr 5, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (

4d63928

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: xinyuxiao <[email protected]>

hongxiayang reviewed Apr 5, 2025

View reviewed changes

simon-mo added a commit that referenced this pull request Apr 6, 2025

Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for in…

296c657

…put queue (#15906)" This reverts commit 651cf0f.

njhill mentioned this pull request Apr 6, 2025

[V1][BugFix] Exit properly if engine core fails during startup #16137

Merged

lulmer pushed a commit to lulmer/vllm that referenced this pull request Apr 7, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (

d96927b

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Louis Ulmer <[email protected]>

sarckk mentioned this pull request Apr 7, 2025

[Bug]: Not able to deploy Llama-4-Scout-17B-16E-Instruct on vllm-openai v0.8.3 #16197

Closed

1 task

xjpang pushed a commit to xjpang/vllm that referenced this pull request Apr 14, 2025

Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for in…

888ef5b

…put queue (vllm-project#15906)" This reverts commit 651cf0f.

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

ZOULQ added a commit to ZOULQ/vllm that referenced this pull request Apr 21, 2025

Revert "Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER socket…

c01904f

…s for input queue (vllm-project#15906)"" This reverts commit 296c657.

heyselbi pushed a commit to red-hat-data-services/vllm that referenced this pull request Apr 23, 2025

Revert "Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER socket…

db0e117

…s for input queue (vllm-project#15906)"" This reverts commit 296c657.

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (

d4f7ba0

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]>

dtrifiro added a commit to red-hat-data-services/vllm that referenced this pull request Apr 30, 2025

Reapply "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for i…

1183fb9

…nput queue (vllm-project#15906)" This reverts commit 296c657.

shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (

a33f405

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]>

shreyankg pushed a commit to shreyankg/vllm that referenced this pull request May 3, 2025

Revert "[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for in…

d4a3dbe

…put queue (vllm-project#15906)" This reverts commit 651cf0f.

RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (

2d55321

vllm-project#15906) Signed-off-by: Nick Hill <[email protected]> Signed-off-by: Mu Huai <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue #15906

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue #15906

Uh oh!

njhill commented Apr 1, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Apr 1, 2025

Uh oh!

Uh oh!

hongxiayang commented Apr 5, 2025

Uh oh!

hongxiayang Apr 5, 2025

Uh oh!

njhill Apr 6, 2025

Uh oh!

njhill Apr 6, 2025

Uh oh!

njhill commented Apr 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue #15906

[V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue #15906

Uh oh!

Conversation

njhill commented Apr 1, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 1, 2025

Uh oh!

Uh oh!

hongxiayang commented Apr 5, 2025

Uh oh!

hongxiayang Apr 5, 2025

Choose a reason for hiding this comment

Uh oh!

njhill Apr 6, 2025

Choose a reason for hiding this comment

Uh oh!

njhill Apr 6, 2025

Choose a reason for hiding this comment

Uh oh!

njhill commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

njhill commented Apr 1, 2025 •

edited by github-actions bot

Loading

njhill commented Apr 6, 2025 •

edited

Loading