Skip to content

Gemma3 Multimodal optimization #404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Gemma3 Multimodal optimization #404

wants to merge 6 commits into from

Conversation

jiminha
Copy link
Contributor

@jiminha jiminha commented Oct 14, 2025

This is to optimize gemma3 multimodal memory/performance.

  • bucket vision tower based on batch bucket to reduce recompile overhead
  • modify merge_multimodal to use torch.where instead of masked_scatter for performance issue
  • add warmup multimodal bucket to precompile vision tower
  • port PT_HPU_SDPA_QKV_SLICE_MODE_FWD feature from vllm-fork v0 : this is necessary to reduce the memory for the longer sequence length.

jiminha and others added 6 commits October 13, 2025 08:03
@jiminha
Initial code for multimodal warmup for Gemma
d982dcf 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Update warmup
8258007 
Signed-off-by: Jimin Ha <[email protected]>
@MohitIntel @jiminha
Overiding the _process_image_input for Gemma3
562e88d 
Signed-off-by: Mohit Deopujari <[email protected]>
@jiminha
Compile error fix on latest vllm merge
153925b 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Update _merge_multimodal_embedding for perf issue in HPU
54227a1 
Signed-off-by: Jimin Ha <[email protected]>
@jiminha
Add FusedSDPA kernel with sliding window support for Gemma3
f21b007 
Reduces memory usage for long sequences by eliminating dual attention
mask creation. Improves capacity from 150 to 400 images with 8K prompts
by avoiding OOM issues.
Limitation: Only available when block_list is None.

Signed-off-by: Jimin Ha <[email protected]>
@github-actions GitHub Actions
Copy link

🚧 CI Blocked

The main CI workflow was not started for the following reason:

Your branch is behind the base branch. Please merge or rebase to get the latest changes.

@jiminha jiminha marked this pull request as draft October 14, 2025 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kzawora-intel kzawora-intel Awaiting requested review from kzawora-intel kzawora-intel is a code owner

@xuechendi xuechendi Awaiting requested review from xuechendi xuechendi is a code owner

@mswiniarsk mswiniarsk Awaiting requested review from mswiniarsk mswiniarsk is a code owner

@adobrzyn adobrzyn Awaiting requested review from adobrzyn adobrzyn is a code owner

@mgawarkiewicz-intel mgawarkiewicz-intel Awaiting requested review from mgawarkiewicz-intel mgawarkiewicz-intel is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

@afierka-intel afierka-intel Awaiting requested review from afierka-intel afierka-intel is a code owner

@michalkuligowski michalkuligowski Awaiting requested review from michalkuligowski michalkuligowski is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jiminha @MohitIntel