feat: add MultimodalParams & putting all multimodal params into it and refactor HyperCLOVAX & Qwen2/2.5-VL #5522
Conversation
yechank-nvidia
changed the title
yechank-nvidia
changed the title
yechank-nvidia
force-pushed
the
multimodal_refactor
branch
from
294077e to
ac0bb0f
Compare
|
/bot run |
|
PR_Github #10815 [ run ] triggered by Bot |
|
PR_Github #10815 [ run ] completed with state |
yechank-nvidia
changed the title
|
/bot run |
|
PR_Github #10820 [ run ] triggered by Bot |
|
PR_Github #10820 [ run ] completed with state |
|
/bot run |
|
PR_Github #10842 [ run ] triggered by Bot |
|
PR_Github #10842 [ run ] completed with state |
yechank-nvidia
force-pushed
the
multimodal_refactor
branch
from
5a466d4 to
a508aee
Compare
|
/bot run |
|
/bot kill |
|
PR_Github #10949 [ run ] triggered by Bot |
|
PR_Github #10950 [ kill ] triggered by Bot |
|
PR_Github #10949 [ run ] completed with state |
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
Signed-off-by: yechank <[email protected]>
yechank-nvidia
force-pushed
the
multimodal_refactor
branch
from
441cc10 to
82cf62e
Compare
|
/bot run |
|
PR_Github #11166 [ run ] triggered by Bot |
|
/bot kill |
Signed-off-by: yechank <[email protected]>
|
PR_Github #11170 [ kill ] triggered by Bot |
|
PR_Github #11166 [ run ] completed with state |
|
PR_Github #11170 [ kill ] completed with state |
Signed-off-by: yechank <[email protected]>
|
/bot run |
|
PR_Github #11173 [ run ] triggered by Bot |
|
PR_Github #11173 [ run ] completed with state |
| if len(multimodal_params) > 0: | ||
| if not DISAGG: | ||
| mm_embeds = self.mm_encoder.forward( | ||
| multimodal_params[:num_context_requests]) |
There was a problem hiding this comment.
Hi @yechank-nvidia, here if some of the ctx requests do not have multimodal items (non-mm request), would it crash?
There was a problem hiding this comment.
It will not. If there is no mm_data found, I retuned input_ids directly so that the len(multimodal_params) = 0, in this case. So that, it skips calling Vision Encoder's forward() and do only LLM-forward.
extra_processed_inputs is None, is multimodal_params is None here
There was a problem hiding this comment.
Right, I mean in one batch of all context requests, len(multimodal_parms) maybe less than num_context_requests right? (as some of the ctx request may pure text prompts)
There was a problem hiding this comment.
I see. Then naively truncating to num_context_requests is not the right solution. Maybe in this case, it will crash. I just assumed there will be case of all text_prompts or all multimodal_prompts.
Let me think how I could handle this.
There was a problem hiding this comment.
Not sure if it is a common or meaningful scenario though...
…d refactor HyperCLOVAX & Qwen2/2.5-VL (NVIDIA#5522) Signed-off-by: yechank <[email protected]> Signed-off-by: Yuxin <[email protected]>
|
Hi, may I ask whether this means Qwen2.5-VL will be supported or have already been? Thanks for your patience! |
|
Hi @Aze1999, you can query either text-only prompts or multimodal-only prompts(meaning has image or video contents). It is fine unless you query mixture of these two. I will file a PR to resolve this issue later! |
|
Hi @yechank-nvidia , big thanks for the fast response! Looking forward to the upcoming PR. Meanwhile, I will try to learn what I can help. |
Description
This PR adds mm_data element so that we could transfer easily mm_related_data.
Also, refactor the Qwen2/2.5-VL & HyperCLOVAX-Vision accordingly.
Especially, moving Qwen2/2.5-VL's encoder into the LLM.
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]Launch build/test pipelines. All previously running jobs will be killed.
--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.md.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.