Skip to content

Conversation

DomBrown
Copy link
Collaborator

@DomBrown DomBrown commented Sep 8, 2025

Previously fixed for NVBug 5461712 in release/1.0 (see #7170), but also apply this to QKNormRoPEAttention

Summary by CodeRabbit

  • New Features

    • Added a user-visible option to disable DeepGEMM in attention and MLP layers. Qwen3 models now default to this setting to improve accuracy. Enables alternative FP8 block-scaling compute paths on supported GPUs.
  • Bug Fixes

    • Improved numerical accuracy for Qwen3 by avoiding DeepGEMM in critical paths.
  • Tests

    • Re-enabled the FP8 block scales test for Qwen3 8B.
    • Updated test metadata to cover additional tracked issues.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

For guidance on mapping tests to stage names, see docs/source/reference/ci-overview.md
and the scripts/test_to_stage_mapping.py helper.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

Previously fixed for NVBug 5461712, but also apply this to QKNormRoPEAttention

Signed-off-by: Dom Brown <[email protected]>
@DomBrown DomBrown self-assigned this Sep 8, 2025
@DomBrown DomBrown requested review from a team as code owners September 8, 2025 11:44
@DomBrown
Copy link
Collaborator Author

DomBrown commented Sep 8, 2025

/bot run

Copy link
Contributor

coderabbitai bot commented Sep 8, 2025

📝 Walkthrough

Walkthrough

Adds a disable_deep_gemm flag across Attention, QKNormRoPEAttention, GatedMLP, Linear, and Qwen3 model wiring. Propagates the flag to underlying Linear layers and adjusts FP8 path selection in Linear to prefer block-scaling GEMM when disable_deep_gemm is set. Re-enables a Qwen3 FP8 test and updates related test metadata.

Changes

Cohort / File(s) Summary
Qwen3 model wiring
tensorrt_llm/_torch/models/modeling_qwen3.py
Sets disable_deep_gemm=True for Qwen3Attention and Qwen3DecoderLayer; forwards to Attention and GatedMLP initializers; adds comments referencing bugs.
Attention modules
tensorrt_llm/_torch/modules/attention.py, tensorrt_llm/_torch/modules/qk_norm_attention.py
Adds disable_deep_gemm parameter to constructors and forwards to internal/base Attention; passes flag to qkv and o Linear projections.
Gated MLP
tensorrt_llm/_torch/modules/gated_mlp.py
Adds disable_deep_gemm to constructor and forwards to internal Linear layers; no forward-path changes.
Linear + FP8 pathing
tensorrt_llm/_torch/modules/linear.py
Adds disable_deep_gemm to Linear; stores on instance; in FP8 apply, selects block-scaling GEMM when use_cute_dsl_blockscaling_mm OR disable_deep_gemm (for SM100); otherwise keeps existing fallback.
Tests and metadata
tests/integration/defs/accuracy/test_llm_api_pytorch.py, tests/integration/test_lists/test-db/l0_b200.yml
Unskips TestQwen3_8B::test_fp8_block_scales; updates YAML comment to reference two nvbugs.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Caller
  participant Linear
  participant FP8Apply as FP8 Apply
  participant Kernels as GPU Kernels

  Caller->>Linear: forward(input)
  activate Linear
  Linear->>FP8Apply: FP8BlockScalesLinearMethod.apply(self, input)
  activate FP8Apply

  Note over FP8Apply: SM == 100?
  alt SM100 and (use_cute_dsl_blockscaling_mm OR disable_deep_gemm) [New OR condition]
    FP8Apply->>Kernels: fp8_quantize_1x128
    FP8Apply->>Kernels: fp8_block_scaling_gemm
  else Otherwise
    FP8Apply->>Kernels: fp8_swap_ab_gemm (existing path)
  end

  deactivate FP8Apply
  FP8Apply-->>Linear: output
  deactivate Linear
  Linear-->>Caller: output
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested reviewers

  • litaotju
  • dcampora
  • MartinMarciniszyn

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (1)
  • Failed to retrieve linked issues from the platform client.
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (4)
tensorrt_llm/_torch/modules/linear.py (2)

1792-1793: Expose new flag in constructor docs and repr.

Minor: consider documenting disable_deep_gemm on Linear (class-level docstring or repr) so it surfaces in logs/config dumps.

Also applies to: 1810-1811


616-631: Optional: add trace to ease profiling.

A lightweight debug trace helps confirm path selection during perf triage.

-            if module.use_cute_dsl_blockscaling_mm or module.disable_deep_gemm:
+            if module.use_cute_dsl_blockscaling_mm or module.disable_deep_gemm:
+                if torch._C._get_tracing_state() is not None:
+                    torch._C._log_api_usage_once("trtllm.linear.fp8_block_scaling_gemm_path")
tensorrt_llm/_torch/models/modeling_qwen3.py (1)

51-54: Make disable_deep_gemm overridable and avoid duplication.

Hardcoding True is fine for the hotfix, but allow an env/config override and define it once per layer to reduce repetition.

@@
-        # Qwen3 has accuracy issues with deep_gemm (see: https://nvbugspro.nvidia.com/bug/5461712
-        # and https://nvbugspro.nvidia.com/bug/5505402)
-        disable_deep_gemm = True
+        # Qwen3 has accuracy issues with deep_gemm (NVBugs: 5461712, 5505402)
+        disable_deep_gemm = (os.getenv("TRTLLM_DISABLE_DEEP_GEMM", "1") == "1")
@@
-        # Qwen3 has accuracy issues with deep_gemm (see: https://nvbugspro.nvidia.com/bug/5461712
-        # and https://nvbugspro.nvidia.com/bug/5505402)
-        disable_deep_gemm = True
+        # Qwen3 has accuracy issues with deep_gemm (NVBugs: 5461712, 5505402)
+        disable_deep_gemm = (os.getenv("TRTLLM_DISABLE_DEEP_GEMM", "1") == "1")

Add at file top:

import os

Also applies to: 86-89

tensorrt_llm/_torch/modules/qk_norm_attention.py (1)

158-159: Plumbing the flag through QKNormRoPEAttention looks correct.

Consider mentioning disable_deep_gemm in the class/init docstring for discoverability.

Also applies to: 182-183

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4a1e138 and 1c01986.

📒 Files selected for processing (7)
  • tensorrt_llm/_torch/models/modeling_qwen3.py (3 hunks)
  • tensorrt_llm/_torch/modules/attention.py (4 hunks)
  • tensorrt_llm/_torch/modules/gated_mlp.py (3 hunks)
  • tensorrt_llm/_torch/modules/linear.py (3 hunks)
  • tensorrt_llm/_torch/modules/qk_norm_attention.py (2 hunks)
  • tests/integration/defs/accuracy/test_llm_api_pytorch.py (0 hunks)
  • tests/integration/test_lists/test-db/l0_b200.yml (1 hunks)
💤 Files with no reviewable changes (1)
  • tests/integration/defs/accuracy/test_llm_api_pytorch.py
🧰 Additional context used
📓 Path-based instructions (3)
**/*.{h,hpp,hh,hxx,cpp,cxx,cc,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Use only spaces, no tabs; indent with 4 spaces.

Files:

  • tensorrt_llm/_torch/modules/linear.py
  • tensorrt_llm/_torch/modules/attention.py
  • tensorrt_llm/_torch/modules/qk_norm_attention.py
  • tensorrt_llm/_torch/models/modeling_qwen3.py
  • tensorrt_llm/_torch/modules/gated_mlp.py
**/*.py

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

**/*.py: Python code must target Python 3.8+.
Indent Python code with 4 spaces; do not use tabs.
Maintain module namespace when importing; prefer 'from package.subpackage import foo' then 'foo.SomeClass()' instead of importing the class directly.
Python filenames should be snake_case (e.g., some_file.py).
Python classes use PascalCase names.
Functions and methods use snake_case names.
Local variables use snake_case; prefix 'k' for variables that start with a number (e.g., k_99th_percentile).
Global variables use upper SNAKE_CASE prefixed with 'G' (e.g., G_MY_GLOBAL).
Constants use upper SNAKE_CASE (e.g., MY_CONSTANT).
Avoid shadowing variables from an outer scope.
Initialize all externally visible members of a class in the constructor.
Prefer docstrings for interfaces that may be used outside a file; comments for in-function or file-local interfaces.
Use Google-style docstrings for classes and functions (Sphinx-parsable).
Document attributes and variables inline so they render under the class/function docstring.
Avoid reflection when a simpler, explicit approach suffices (e.g., avoid dict(**locals()) patterns).
In try/except, catch the most specific exceptions possible.
For duck-typing try/except, keep the try body minimal and use else for the main logic.

Files:

  • tensorrt_llm/_torch/modules/linear.py
  • tensorrt_llm/_torch/modules/attention.py
  • tensorrt_llm/_torch/modules/qk_norm_attention.py
  • tensorrt_llm/_torch/models/modeling_qwen3.py
  • tensorrt_llm/_torch/modules/gated_mlp.py
**/*.{cpp,cxx,cc,h,hpp,hh,hxx,cu,cuh,py}

📄 CodeRabbit inference engine (CODING_GUIDELINES.md)

Prepend the NVIDIA Apache-2.0 copyright header with current year to the top of all source files (e.g., .cpp, .h, .cu, .py).

Files:

  • tensorrt_llm/_torch/modules/linear.py
  • tensorrt_llm/_torch/modules/attention.py
  • tensorrt_llm/_torch/modules/qk_norm_attention.py
  • tensorrt_llm/_torch/models/modeling_qwen3.py
  • tensorrt_llm/_torch/modules/gated_mlp.py
🧠 Learnings (1)
📚 Learning: 2025-07-28T17:06:08.621Z
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

  • tests/integration/test_lists/test-db/l0_b200.yml
🧬 Code graph analysis (1)
tensorrt_llm/_torch/models/modeling_qwen3.py (1)
tensorrt_llm/_torch/modules/gated_mlp.py (1)
  • GatedMLP (19-182)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Pre-commit Check
🔇 Additional comments (7)
tests/integration/test_lists/test-db/l0_b200.yml (1)

40-40: TestQwen3_8B::test_fp8_block_scales[latency] is present and not skipped. Verified the def test_fp8_block_scales with ids=["latency"] in TestQwen3_8B has no skip decorators.

tensorrt_llm/_torch/modules/linear.py (1)

616-631: Correct SM100 fallback away from DeepGEMM.

Routing to block-scaling GEMM when disable_deep_gemm is set is clear and localized. LGTM.

tensorrt_llm/_torch/models/modeling_qwen3.py (1)

67-68: Propagation into Attention and MLP is correct.

Forwarding the flag down to Attention/GatedMLP aligns with the Linear path selection. LGTM.

Also applies to: 96-97

tensorrt_llm/_torch/modules/gated_mlp.py (2)

74-76: Flag propagation into both Linear submodules is consistent.

No issues spotted.

Also applies to: 94-96


21-35: All GatedMLP instantiations use keyword-only arguments
Adding the new disable_deep_gemm keyword parameter is backward-compatible with every existing call site.

tensorrt_llm/_torch/modules/attention.py (2)

119-140: Good API extension and docstring update.

Adding disable_deep_gemm at the tail with a default avoids breaking callers; docstring entry is clear.


220-223: Propagation into qkv and o projections is correct.

Matches the intended Linear behavior switch. LGTM.

Also applies to: 238-241

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18041 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18041 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13518 completed with status: 'FAILURE'

@DomBrown
Copy link
Collaborator Author

DomBrown commented Sep 8, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18061 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18061 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13535 completed with status: 'FAILURE'

@DomBrown
Copy link
Collaborator Author

DomBrown commented Sep 8, 2025

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18076 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18076 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13549 completed with status: 'FAILURE'

@DomBrown
Copy link
Collaborator Author

DomBrown commented Sep 8, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18097 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #18097 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #13561 completed with status: 'SUCCESS'

@yuxianq yuxianq changed the title [https://nvbugs/55054402] [fix] Disable deep_gemm for Qwen3 QKNormRoPEAttention and Linear layers due to accuracy issues [https://nvbugs/5505402] [fix] Disable deep_gemm for Qwen3 QKNormRoPEAttention and Linear layers due to accuracy issues Sep 9, 2025
Copy link
Collaborator

@QiJune QiJune left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@DomBrown DomBrown merged commit fc9d426 into NVIDIA:main Sep 10, 2025
9 of 11 checks passed
Wong4j pushed a commit to Wong4j/TensorRT-LLM that referenced this pull request Sep 20, 2025
…Attention and Linear layers due to accuracy issues (NVIDIA#7616)

Signed-off-by: Dom Brown <[email protected]>
MrGeva pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request Sep 21, 2025
…Attention and Linear layers due to accuracy issues (NVIDIA#7616)

Signed-off-by: Dom Brown <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants