forked from NVIDIA/TensorRT-LLM
-
Notifications
You must be signed in to change notification settings - Fork 0
[None][autodeploy] Add quantization_source to factory interface #100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2f95386
to
bb1cb83
Compare
da1852f
to
2dd9839
Compare
lucaslie
reviewed
Jul 31, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks mostly good to me. Just a few comments regarding the reader interface but we are almost there :) Please work with @suyoggupta while I am gone to get it merged
2dd9839
to
8361bd6
Compare
a456cd7
to
9b1f417
Compare
Signed-off-by: Frida Hou <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Frida Hou <[email protected]> remove quant_source getter Signed-off-by: Frida Hou <[email protected]>
Signed-off-by: Frida Hou <[email protected]> minor Signed-off-by: Frida Hou <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Frida Hou <[email protected]> revert Llama4 FP8 patch Signed-off-by: Frida Hou <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Frida Hou <[email protected]>
Signed-off-by: Frida Hou <[email protected]>
9b1f417
to
ea15cb4
Compare
suyoggupta
reviewed
Aug 6, 2025
tensorrt_llm/_torch/auto_deploy/transform/library/quantization.py
Outdated
Show resolved
Hide resolved
suyoggupta
approved these changes
Aug 6, 2025
ea15cb4
to
e8c9e28
Compare
…o the transformation, spilit transformation into config and graph based Signed-off-by: Frida Hou <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Frida Hou <[email protected]> move quantization target collection as a transform Signed-off-by: Frida Hou <[email protected]> minor Signed-off-by: Frida Hou <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Frida Hou <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Frida Hou <[email protected]> minor: remove QuantizationTarget Signed-off-by: Frida Hou <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Frida Hou <[email protected]> update transform docstring Signed-off-by: Frida Hou <[email protected]> fix unit test Signed-off-by: Frida Hou <[email protected]>
e8c9e28
to
6c03e65
Compare
Signed-off-by: Frida Hou <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 8, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 8, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 9, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 9, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> address feedback Signed-off-by: Frida Hou <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 11, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> address feedback Signed-off-by: Frida Hou <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 11, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> address feedback Signed-off-by: Frida Hou <[email protected]>
Fridah-nv
added a commit
that referenced
this pull request
Aug 11, 2025
* Move quant_config handling to _load_quantization_config Signed-off-by: Fridah-nv <[email protected]> move kv_cache_dtype into _quant_config in hf factory Signed-off-by: Fridah-nv <[email protected]> remove quant_source getter Signed-off-by: Fridah-nv <[email protected]> * add QuantConfigReader class Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:Llama4 FP8 for BMM testing Signed-off-by: Fridah-nv <[email protected]> revert Llama4 FP8 patch Signed-off-by: Fridah-nv <[email protected]> move _quant_config into QuantConfigReader Signed-off-by: Fridah-nv <[email protected]> * move quantize and quantize_moe to the end of pattern matcher Signed-off-by: Fridah-nv <[email protected]> * delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based Signed-off-by: Fridah-nv <[email protected]> have quantConfigReader return the dtype for NVFP4 Signed-off-by: Fridah-nv <[email protected]> move quantization target collection as a transform Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> tmp:hacky fix modelopt graph based path quantizer loading Signed-off-by: Fridah-nv <[email protected]> fix rebase quantization to BaseTransform Signed-off-by: Fridah-nv <[email protected]> minor: remove QuantizationTarget Signed-off-by: Fridah-nv <[email protected]> rm tmp fix, rebase, minor update on interface Signed-off-by: h-guo18 <[email protected]> Signed-off-by: Fridah-nv <[email protected]> update transform docstring Signed-off-by: Fridah-nv <[email protected]> fix unit test Signed-off-by: Fridah-nv <[email protected]> * move quantization to end of transformations Signed-off-by: Fridah-nv <[email protected]> --------- Signed-off-by: Fridah-nv <[email protected]> minor Signed-off-by: Fridah-nv <[email protected]> address feedback Signed-off-by: Frida Hou <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Test Coverage
Before:
TP=1
After:
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...
Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]
to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]
Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id
(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test
(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast
(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test
(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"
(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"
(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"
(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test
(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test
(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test
(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge
(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"
(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log
(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug
(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-list
parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.md
and the
scripts/test_to_stage_mapping.py
helper.kill
kill
Kill all running builds associated with pull request.
skip
skip --comment COMMENT
Skip testing for latest commit on pull request.
--comment "Reason for skipping build/test"
is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipeline
Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.