Skip to content

Conversation

xinhe-nv
Copy link
Collaborator

@xinhe-nv xinhe-nv commented Aug 17, 2025

Remove closed bugs from waives.txt

Summary by CodeRabbit

  • Tests
    • Re-enabled many previously-waived integration tests to broaden automated coverage.
    • Added targeted skips for a set of Phi FP8/BF16 LoRA tests and two Llama-related tests to avoid known issues.
    • Added extra post-blackwell skip markers on two Gemma parameterized tests to reduce flakiness while keeping overall verification expanded.

Copy link
Contributor

coderabbitai bot commented Aug 17, 2025

📝 Walkthrough

Walkthrough

Removed and added SKIP entries in the integration waives list; added extra @skip_post_blackwell decorators (duplicated) above two Gemma tests in test_gemma.py; no signature or functional test logic changes.

Changes

Cohort / File(s) Change summary
Waives list updates
tests/integration/test_lists/waives.txt
Removed many previous SKIP entries across Gemma, multimodal, e2e quickstart multimodal, and Phi4MM-related tests; added new SKIP entries including several Phi FP8/BF16 LoRA tests (Phi-3-mini-128k-instruct, Phi-3-small-128k-instruct, Phi-3.5-mini-instruct, Phi-4-mini-instruct) and two llama tests (llm_api_lookahead_decoding_1gpu, llm_llama_v1_2gpu_summary).
Gemma test decorators
tests/integration/defs/examples/test_gemma.py
Inserted an additional @skip_post_blackwell decorator (resulting in two consecutive instances) above test_llm_hf_gemma_quantization_1gpu_vswa and test_llm_gemma_1gpu_summary_vswa; parameterization and function signatures unchanged.

Sequence Diagram(s)

sequenceDiagram
  participant TR as Test Runner
  participant D as @skip_post_blackwell
  participant T as Parametrized Test

  TR->>T: Discover test (parametrized)
  TR->>D: Evaluate decorator(s) for post-blackwell
  alt any decorator indicates skip
    D-->>TR: Return skip
    TR->>T: Mark as skipped (no invocations)
  else none indicate skip
    TR->>T: Execute parameterized invocations
  end
  Note right of D: Two consecutive decorators present for some tests
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • crazydemo
  • LarryXFly

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai or @coderabbitai title anywhere in the PR title to generate the title automatically.

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@xinhe-nv xinhe-nv force-pushed the user/qa/unwaive_20250817 branch from c3d2174 to 1bcc157 Compare August 18, 2025 01:23
@xinhe-nv xinhe-nv marked this pull request as ready for review August 18, 2025 01:24
@xinhe-nv xinhe-nv enabled auto-merge (squash) August 18, 2025 01:24
@xinhe-nv
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #15552 [ run ] triggered by Bot

@xinhe-nv xinhe-nv force-pushed the user/qa/unwaive_20250817 branch 2 times, most recently from 028a7c1 to 4ba2b30 Compare August 18, 2025 07:15
@tensorrt-cicd
Copy link
Collaborator

PR_Github #15552 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #11713 completed with status: 'SUCCESS'

Signed-off-by: xinhe-nv <[email protected]>
Signed-off-by: Xin He (SW-GPU) <[email protected]>
@xinhe-nv xinhe-nv force-pushed the user/qa/unwaive_20250817 branch from 4ba2b30 to 879893d Compare August 19, 2025 07:55
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/integration/test_lists/waives.txt (1)

315-318: Validate Phi FP8+BF16 LoRA waivers

  • ✅ The test_phi_fp8_with_bf16_lora function is defined in tests/integration/defs/examples/test_phi.py (line 382).
  • ✅ All four parametrizations (Phi-3-mini-128k-instruct, Phi-3-small-128k-instruct, Phi-3.5-mini-instruct, Phi-4-mini-instruct) are present in that file.
  • ✅ No duplicate waiver entries were found in tests/integration/test_lists/waives.txt.
  • ⚠️ Confirm whether failures occur across all platforms or only on specific ones. If they’re limited (e.g. only on CUDA or ROCm), switch from a global skip to a platform-scoped prefix, e.g.
    full:CUDA/examples/test_phi.py::test_phi_fp8_with_bf16_lora[...] SKIP (…)
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 4ba2b30 and 879893d.

📒 Files selected for processing (2)
  • tests/integration/defs/examples/test_gemma.py (1 hunks)
  • tests/integration/test_lists/waives.txt (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/integration/defs/examples/test_gemma.py
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-07-28T17:06:08.621Z
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.

Applied to files:

  • tests/integration/test_lists/waives.txt
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Pre-commit Check

@LarryXFly LarryXFly disabled auto-merge August 19, 2025 08:01
@LarryXFly LarryXFly merged commit 2c86cee into NVIDIA:main Aug 19, 2025
4 checks passed
@xinhe-nv xinhe-nv deleted the user/qa/unwaive_20250817 branch August 19, 2025 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants