test: [CI] Add failed cases into waives.txt #6333
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📝 Walkthrough## Walkthrough
Decorators to conditionally skip tests (`@skip_pre_hopper` and `pytest.mark.skip_less_device_memory`) were added to specific test cases in the integration test suite. Several FP8 prequantized model tests are now skipped under certain conditions, and two multimodal model tests are skipped if device memory is insufficient. Two new test cases were added to a test list. Additionally, test skip entries were added for known bugs in the multimodal test suite.
## Changes
| File(s) | Change Summary |
|------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| tests/integration/defs/accuracy/test_cli_flow.py | Replaced `@skip_pre_ada` with `@skip_pre_hopper` on multiple FP8 prequantized test methods. |
| tests/integration/defs/accuracy/test_llm_api_pytorch.py | Added `@skip_pre_hopper` decorator to multiple FP8 prequantized test methods across classes. |
| tests/integration/defs/test_e2e.py | Updated two parameterized test cases to use `pytest.param` with `skip_less_device_memory(80000)`. |
| tests/integration/test_lists/qa/llm_sanity_test.txt | Added two new test cases for `gemma-3-27b-it-gemma` with image input enabled and disabled. |
| tests/integration/test_lists/waives.txt | Added two skip entries for multimodal test cases referencing known bugs (nvbugs/5401114, 5414909).|
## Sequence Diagram(s)
```mermaid
sequenceDiagram
participant Tester
participant Pytest
participant Decorator
Tester->>Pytest: Run integration tests
Pytest->>Decorator: Evaluate @skip_pre_hopper/@skip_less_device_memory
alt Skip condition met
Decorator-->>Pytest: Mark test as skipped
Pytest-->>Tester: Report test skipped
else Condition not met
Pytest->>Tester: Run test normally
endEstimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Suggested reviewers
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/integration/test_lists/waives.txt (1)
443-443: Nit: keep bug‐URL hostnames consistentElsewhere we sometimes use the full host
https://nvbugspro.nvidia.com/bug/…. Using a uniform hostname (nvbugsprovsnvbugs) simplifies grep-based analytics on the waive list. Optional but worth considering for future entries.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
tests/integration/test_lists/waives.txt(1 hunks)
🧰 Additional context used
🧠 Learnings (1)
tests/integration/test_lists/waives.txt (1)
Learnt from: yiqingy0
PR: #5198
File: jenkins/mergeWaiveList.py:0-0
Timestamp: 2025-07-22T08:33:49.109Z
Learning: In the TensorRT-LLM waive list merging system, removed lines are always located at the end of the merge waive lists, which is why the mergeWaiveList.py script uses reverse traversal - it's an optimization for this specific domain constraint.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (1)
tests/integration/test_lists/waives.txt (1)
443-443: Entry format is valid – change acceptedThe new skip line follows existing conventions (test node,
SKIP, reason in parentheses). No duplicate ofTestLlama3_2_1B::test_fp8_prequantizedexists above, so the entry is safe to merge.
xinhe-nv
force-pushed
the
user/qa/post_update_waive_20250724_LLM_FUNCTION_TEST_1037
branch
from
8f75f6c to
56b4f65
Compare
xinhe-nv
force-pushed
the
user/qa/post_update_waive_20250724_LLM_FUNCTION_TEST_1037
branch
from
56b4f65 to
fc7ae80
Compare
xinhe-nv
force-pushed
the
user/qa/post_update_waive_20250724_LLM_FUNCTION_TEST_1037
branch
2 times, most recently
from
9060c39 to
654cc38
Compare
xinhe-nv
changed the title
|
/bot run |
|
PR_Github #12947 [ run ] triggered by Bot |
|
PR_Github #12947 [ run ] completed with state |
Signed-off-by: Xin He (SW-GPU) <[email protected]>
xinhe-nv
force-pushed
the
user/qa/post_update_waive_20250724_LLM_FUNCTION_TEST_1037
branch
from
654cc38 to
610376f
Compare
LarryXFly
approved these changes
Jul 25, 2025
xinhe-nv
deleted the
user/qa/post_update_waive_20250724_LLM_FUNCTION_TEST_1037
branch
NVShreyas
pushed a commit
to NVShreyas/TensorRT-LLM
that referenced
this pull request
Jul 28, 2025
Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Shreyas Misra <[email protected]>
Ransiki
pushed a commit
to Ransiki/TensorRT-LLM
that referenced
this pull request
Jul 29, 2025
Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Ransiki Zhang <[email protected]>
lancelly
pushed a commit
to lancelly/TensorRT-LLM
that referenced
this pull request
Aug 6, 2025
Signed-off-by: Xin He (SW-GPU) <[email protected]> Signed-off-by: Lanyu Liao <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
skip fp8 tests on A100.
skip mistral-small-3.1-24b and gemma-3-27b on L20
test report https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Function-Pipelines/job/LLM_FUNCTION_TEST_DEBUG/1463/allure/
https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/view/TRT-LLM-Function-Pipelines/job/LLM_FUNCTION_TEST_DEBUG/1464/allure/
Summary by CodeRabbit