Skip to content

Conversation

Fridah-nv
Copy link

@Fridah-nv Fridah-nv commented Jul 22, 2025

@coderabbitai summary

Description

Test Coverage

  • Unit tests
  • E2E tested FP8 models: config based, graph based, Moe(Mixtral), BMM(llama4)
  • Perf results:
    Before:

TP=1

Request Throughput (req/sec):                     6.9926
Total Output Throughput (tokens/sec):             895.0499
Total Token Throughput (tokens/sec):              1790.0999
Total Latency (ms):                               143008.7812
Average request latency (ms):                     75925.5988
Per User Output Throughput [w/ ctx] (tps/user):   2.8372
Per GPU Output Throughput (tps/gpu):              895.0499

After:

Request Throughput (req/sec):                     7.0424
Total Output Throughput (tokens/sec):             901.4267
Total Token Throughput (tokens/sec):              1802.8533
Total Latency (ms):                               141997.1308
Average request latency (ms):                     74739.9443
Per User Output Throughput [w/ ctx] (tps/user):   2.8697
Per GPU Output Throughput (tps/gpu):              901.4267

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

@Fridah-nv Fridah-nv marked this pull request as draft July 22, 2025 19:51
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch 2 times, most recently from 2f95386 to bb1cb83 Compare July 24, 2025 23:25
@Fridah-nv Fridah-nv changed the base branch from feat/ad-2025-07-07 to feat/ad-2025-07-22 July 24, 2025 23:25
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch 6 times, most recently from da1852f to 2dd9839 Compare July 31, 2025 07:51
Copy link
Collaborator

@lucaslie lucaslie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks mostly good to me. Just a few comments regarding the reader interface but we are almost there :) Please work with @suyoggupta while I am gone to get it merged

@Fridah-nv Fridah-nv self-assigned this Aug 5, 2025
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch from 2dd9839 to 8361bd6 Compare August 5, 2025 18:23
@Fridah-nv Fridah-nv changed the base branch from feat/ad-2025-07-22 to feat/ad-arch-2025-08-01 August 5, 2025 18:24
@Fridah-nv Fridah-nv changed the title [Draft]add quantization_source to factory interface [None][AutoDeploy]add quantization_source to factory interface Aug 5, 2025
@Fridah-nv Fridah-nv marked this pull request as ready for review August 5, 2025 19:45
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch from a456cd7 to 9b1f417 Compare August 5, 2025 19:47
Signed-off-by: Frida Hou <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Frida Hou <[email protected]>

remove quant_source getter

Signed-off-by: Frida Hou <[email protected]>
Signed-off-by: Frida Hou <[email protected]>

minor

Signed-off-by: Frida Hou <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Frida Hou <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Frida Hou <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Frida Hou <[email protected]>
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch from 9b1f417 to ea15cb4 Compare August 5, 2025 19:50
@Fridah-nv Fridah-nv changed the title [None][AutoDeploy]add quantization_source to factory interface [None]add quantization_source to factory interface Aug 5, 2025
@Fridah-nv Fridah-nv changed the title [None]add quantization_source to factory interface [None][auto deploy]add quantization_source to factory interface Aug 5, 2025
@Fridah-nv Fridah-nv changed the title [None][auto deploy]add quantization_source to factory interface [None][autodeploy] Add quantization_source to factory interface Aug 5, 2025
@Fridah-nv Fridah-nv requested a review from suyoggupta August 5, 2025 19:57
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch from ea15cb4 to e8c9e28 Compare August 8, 2025 00:33
…o the transformation, spilit transformation into config and graph based

Signed-off-by: Frida Hou <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Frida Hou <[email protected]>

move quantization target collection as a transform

Signed-off-by: Frida Hou <[email protected]>

minor

Signed-off-by: Frida Hou <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Frida Hou <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Frida Hou <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Frida Hou <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Frida Hou <[email protected]>

update transform docstring

Signed-off-by: Frida Hou <[email protected]>

fix unit test

Signed-off-by: Frida Hou <[email protected]>
@Fridah-nv Fridah-nv force-pushed the user/fridah/quant-refactor branch from e8c9e28 to 6c03e65 Compare August 8, 2025 01:05
@Fridah-nv Fridah-nv merged commit ab752f1 into feat/ad-arch-2025-08-01 Aug 8, 2025
3 checks passed
Fridah-nv added a commit that referenced this pull request Aug 8, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 8, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 9, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 9, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

address feedback

Signed-off-by: Frida Hou <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 11, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

address feedback

Signed-off-by: Frida Hou <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 11, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

address feedback

Signed-off-by: Frida Hou <[email protected]>
Fridah-nv added a commit that referenced this pull request Aug 11, 2025
* Move quant_config handling to _load_quantization_config

Signed-off-by: Fridah-nv <[email protected]>

move kv_cache_dtype into _quant_config in hf factory

Signed-off-by: Fridah-nv <[email protected]>

remove quant_source getter

Signed-off-by: Fridah-nv <[email protected]>

* add QuantConfigReader class

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:Llama4 FP8 for BMM testing

Signed-off-by: Fridah-nv <[email protected]>

revert Llama4 FP8 patch

Signed-off-by: Fridah-nv <[email protected]>

move _quant_config into QuantConfigReader

Signed-off-by: Fridah-nv <[email protected]>

* move quantize and quantize_moe to the end of pattern matcher

Signed-off-by: Fridah-nv <[email protected]>

* delegate quant_config processing to QuantConfigReader and pass read to the transformation, spilit transformation into config and graph based

Signed-off-by: Fridah-nv <[email protected]>

have quantConfigReader return the dtype for NVFP4

Signed-off-by: Fridah-nv <[email protected]>

move quantization target collection as a transform

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

tmp:hacky fix modelopt graph based path quantizer loading

Signed-off-by: Fridah-nv <[email protected]>

fix rebase quantization to BaseTransform

Signed-off-by: Fridah-nv <[email protected]>

minor: remove QuantizationTarget

Signed-off-by: Fridah-nv <[email protected]>

rm tmp fix, rebase, minor update on interface

Signed-off-by: h-guo18 <[email protected]>
Signed-off-by: Fridah-nv <[email protected]>

update transform docstring

Signed-off-by: Fridah-nv <[email protected]>

fix unit test

Signed-off-by: Fridah-nv <[email protected]>

* move quantization to end of transformations

Signed-off-by: Fridah-nv <[email protected]>

---------

Signed-off-by: Fridah-nv <[email protected]>

minor

Signed-off-by: Fridah-nv <[email protected]>

address feedback

Signed-off-by: Frida Hou <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants