[v1] Add Whisper model support (encoder-decoder) #21088

russellb · 2025-07-17T02:13:06Z

v1: Add Whisper encoder-decoder model support

Implements Whisper mdoel support in the V1 engine. Key changes include:

Add encoder-decoder architecture support with cross-attention KV cache management
Add CrossAttentionManager and CrossAttentionSpec for encoder-decoder KV cache
Update scheduler to handle cross-attention block allocation and disable prefix caching
Modify GPU model runner for encoder input processing and attention metadata
Disable BART tests/examples (Whisper-only support for now)

This closes a major feature gap between V0 and V1, enabling Whisper transcription
in the new engine architecture while maintaining backward compatibility.

Related to V0 deprecation (#18571) and 2025 Q3 roadmap (#20336).

Closes #12761

Signed-off-by: Russell Bryant [email protected]
Co-authored-by: NickLucche [email protected]

Follow-up TODO items

utilize encoder cache
remove custom cross-attention slot mapping calculation and integrate into the proper abstraction
clean up hard coded assumptions about whisper / multi-modal

mergify · 2025-07-17T02:13:48Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @russellb.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

gemini-code-assist

Code Review

This is a significant and well-structured pull request that adds Whisper (encoder-decoder) model support to vLLM's V1 engine. The changes are comprehensive, touching on the attention backend, KV cache management, scheduler, and GPU model runner to accommodate the new architecture.

I've identified one critical issue in _build_encoder_attn_metadata where a missing else block could lead to a size mismatch and a runtime error. I've provided a code suggestion to fix this potential bug. Other than that, the implementation looks solid and correctly integrates encoder-decoder support into the existing V1 framework. Great work on this complex feature!

vllm/v1/worker/gpu_model_runner.py

github-actions · 2025-07-17T02:24:48Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

DarkLight1337 · 2025-07-17T02:51:14Z

This is already some work to support encoder-decoder models:

Can you coordinate with @maxdebayser to avoid duplicate work?

maxdebayser · 2025-07-17T14:31:18Z

Yeah, I've been talking with @russellb as there are a few overlapping points in our PRs for example disabling prefix caching and chunked prefill.
Currently in my PR I'm not disabling the KV cache entirely because functionally it makes no difference for the encoder attention. So I can keep the diff small. But I do want to test if removing the KV cache will have a performance improvement for encoder models

vllm/v1/attention/backends/flash_attn.py

vllm/v1/core/single_type_kv_cache_manager.py

russellb · 2025-07-17T14:44:09Z

This is already some work to support encoder-decoder models:

Add support for encoder embedding models #19988

[Bug]: RuntimeError: NCCL error: unhandled cuda error #20226

Can you coordinate with @maxdebayser to avoid duplicate work?

Yep, we're in contact.

Did you mean to link something different than #20226?

Roughly though, Max had worked on encoder-only support, and I was doing encoder-decoder, which is mostly a superset of encoder-only changes, though I haven't actually tested any encoder-only models with my branch yet.

vllm/v1/attention/backends/flash_attn.py

russellb · 2025-07-17T18:35:17Z

follow-up on next steps and collaboration with @maxdebayser

We're going to combine our work and try to land it all in a few stages.

PR 1) Combine parts of his encoder-only PR (#19988) with the encoder-without-kv-cache changes in this branch. That will be a new jointly-authored PR that will cover encoder-only attention.

PR 2) Update this PR with what's left to make Whisper / encoder-decoder work. That includes some Whisper model changes and a bunch of changes to support cross-attention (encoder-decoder type).

PR 3) Add the last parts of Max's original PR, which supports token_type_ids to run the bert classifier models that need them.

NickLucche

nice one!

vllm/v1/attention/backends/flash_attn.py

vllm/v1/core/kv_cache_coordinator.py

russellb · 2025-07-18T20:48:24Z

I got this caught up with main with all conflicts resolved, but I haven't addressed feedback received so far.

mergify · 2025-07-19T09:39:16Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @russellb.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Add support for encoder models such as BERT which don't support a KV cache due to the non-causal attention. Since the KV Cache Spec is used to build the attention metadata for decoder models, this PR initializes the attention metadata builds for encoder-only models directly from the layers and adds a function to build the attention metadata. This PR combines elements of PRs vllm-project#21088 and vllm-project#19988 Summary of changes: **Flash Attention Backend:** - Implement encoder self-attention support without using KV cache **Scheduler:** - Disable chunked prefill for models without KV cache **GPU Model Runner:** - Implement encoder-only attention metadata building for self-attention Related to: - V0 deprecation: vllm-project#18571 - 2025 Q3 roadmap: vllm-project#20336 Signed-off-by: Max de Bayser <[email protected]> Co-authored-by: Russell Bryant <[email protected]>

Signed-off-by: Russell Bryant <[email protected]>

heheda12345

LGTM! Thanks for your patience for iterating on this. It is really helpful for v0 deprecation.

vllm/attention/layer.py

vllm/v1/core/encoder_cache_manager.py

Signed-off-by: Russell Bryant <[email protected]>

russellb · 2025-09-10T18:08:35Z

LGTM! Thanks for your patience for iterating on this. It is really helpful for v0 deprecation.

and thank you for all of the time invested in review! The end result is a lot cleaner because of your feedback.

Signed-off-by: Russell Bryant <[email protected]> Co-authored-by: NickLucche <[email protected]>

russellb mentioned this pull request Jul 16, 2025

[Roadmap] vLLM Roadmap Q3 2025 #20336

Open

mergify bot added v1 needs-rebase labels Jul 17, 2025

gemini-code-assist bot reviewed Jul 17, 2025

View reviewed changes

vllm/v1/worker/gpu_model_runner.py Outdated Show resolved Hide resolved

maxdebayser reviewed Jul 17, 2025

View reviewed changes

vllm/v1/attention/backends/flash_attn.py Outdated Show resolved Hide resolved

maxdebayser reviewed Jul 17, 2025

View reviewed changes

vllm/v1/core/single_type_kv_cache_manager.py Outdated Show resolved Hide resolved

LucasWilkinson reviewed Jul 17, 2025

View reviewed changes

vllm/v1/attention/backends/flash_attn.py Outdated Show resolved Hide resolved

russellb force-pushed the v1-whisper branch 3 times, most recently from 96be9ad to 4da8b7c Compare July 17, 2025 19:27

NickLucche requested changes Jul 18, 2025

View reviewed changes

russellb force-pushed the v1-whisper branch 3 times, most recently from 16f557d to a9e3459 Compare July 18, 2025 20:46

mergify bot added documentation Improvements or additions to documentation and removed needs-rebase labels Jul 18, 2025

russellb force-pushed the v1-whisper branch from a9e3459 to 8b080c3 Compare July 18, 2025 20:47

russellb force-pushed the v1-whisper branch 2 times, most recently from 87d9bfa to f62a66e Compare July 18, 2025 21:00

mergify bot added the needs-rebase label Jul 19, 2025

maxdebayser mentioned this pull request Jul 20, 2025

Support encoder-only models without KV-Cache #21270

Merged

russellb added 15 commits September 10, 2025 17:05

Move max_seq_len override into CrossAttentionBuilder

11b9b9e

Signed-off-by: Russell Bryant <[email protected]>

Move seq_lens / seq_lens_cpu overrides into CrossAttentionBuilder

a875bfc

Signed-off-by: Russell Bryant <[email protected]>

Revert unnecessary code move to reduce diff size

8c7176f

Signed-off-by: Russell Bryant <[email protected]>

further simplification of slot_mappings for cross attn

7304b3e

Signed-off-by: Russell Bryant <[email protected]>

move slot mapping calculation back into CrossAttentionBuilder

83fd244

Signed-off-by: Russell Bryant <[email protected]>

improve how we get the number of blocks needed

04cf403

Signed-off-by: Russell Bryant <[email protected]>

make python loop more efficient

431db03

Signed-off-by: Russell Bryant <[email protected]>

remove leftover docstring addition

279c0b0

Signed-off-by: Russell Bryant <[email protected]>

Restore TODO that was accidentally removed

d466655

Signed-off-by: Russell Bryant <[email protected]>

remove variables no longer needed

bc7277e

Signed-off-by: Russell Bryant <[email protected]>

Ensure AttentionMetadataBuilder subclasses call parent constructor

1c28542

Signed-off-by: Russell Bryant <[email protected]>

remove old param from docstring

4b31447

Signed-off-by: Russell Bryant <[email protected]>

move MultiHeadAttention changes into whisper.py

8305707

Signed-off-by: Russell Bryant <[email protected]>

use vllm.utils.cdiv

3c10b1f

Signed-off-by: Russell Bryant <[email protected]>

minor refactoring

7444d37

Signed-off-by: Russell Bryant <[email protected]>

russellb force-pushed the v1-whisper branch from 5bf5479 to 7444d37 Compare September 10, 2025 17:07

russellb requested a review from NickLucche September 10, 2025 17:44

heheda12345 approved these changes Sep 10, 2025

View reviewed changes

vllm/attention/layer.py Outdated Show resolved Hide resolved

vllm/v1/core/encoder_cache_manager.py Show resolved Hide resolved

revert unnecessary changes left over

4afbdcb

Signed-off-by: Russell Bryant <[email protected]>

heheda12345 enabled auto-merge (squash) September 10, 2025 18:10

simon-mo disabled auto-merge September 10, 2025 20:53

simon-mo merged commit 37e8182 into vllm-project:main Sep 10, 2025
69 of 72 checks passed

DarkLight1337 mentioned this pull request Sep 11, 2025

[Bugfix] Fix incorrect import of CacheConfig #24631

Merged

5 tasks

skyloevil pushed a commit to skyloevil/vllm that referenced this pull request Sep 13, 2025

[v1] Add Whisper model support (encoder-decoder) (vllm-project#21088)

93899a9

Signed-off-by: Russell Bryant <[email protected]> Co-authored-by: NickLucche <[email protected]>

FeiDaLI pushed a commit to FeiDaLI/vllm that referenced this pull request Sep 25, 2025

[v1] Add Whisper model support (encoder-decoder) (vllm-project#21088)

c0913b6

Signed-off-by: Russell Bryant <[email protected]> Co-authored-by: NickLucche <[email protected]>

Uh oh!

[v1] Add Whisper model support (encoder-decoder) #21088

[v1] Add Whisper model support (encoder-decoder) #21088

Conversation

russellb commented Jul 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Jul 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

github-actions bot commented Jul 17, 2025

Uh oh!

DarkLight1337 commented Jul 17, 2025

Uh oh!

maxdebayser commented Jul 17, 2025

Uh oh!

Uh oh!

Uh oh!

russellb commented Jul 17, 2025

Uh oh!

Uh oh!

russellb commented Jul 17, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

russellb commented Jul 18, 2025

Uh oh!

mergify bot commented Jul 19, 2025

Uh oh!

heheda12345 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

russellb commented Sep 10, 2025

Uh oh!

Uh oh!

Uh oh!

russellb commented Jul 17, 2025 •

edited by github-actions bot

Loading