[https://nvbugs/5457489][fix] unwaive some tests #6991
Conversation
📝 WalkthroughWalkthroughUpdated test waivers (SKIP entries) and reduced timeouts for specific latency tests; test_nvfp4 now constructs LLM with max_batch_size=32 and runs an MMLU evaluation inside the existing LLM context before GSM8K. No public APIs or data-structure signatures changed. Changes
Sequence Diagram(s)sequenceDiagram
participant Runner as TestRunner
participant LLM as LLM(context)
participant MMLU as MMLU_Task
participant GSM8K as GSM8K_Task
Runner->>LLM: with LLM(..., max_batch_size=32) as llm
LLM-->>Runner: llm ready
Runner->>MMLU: create MMLU task
Runner->>MMLU: mmlu.evaluate(llm)
MMLU-->>Runner: MMLU results
Runner->>GSM8K: create GSM8K task
Runner->>GSM8K: gsm8k.evaluate(llm)
GSM8K-->>Runner: GSM8K results
Runner->>LLM: exit context
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
|
/bot run |
|
/bot run --disable-fail-fast |
byshiue
changed the title
|
/bot run --stage-list "B200_PCIe-PyTorch-1, B200_PCIe-PyTorch-2, B200_PCIe-PyTorch-3" |
|
/bot run --stage-list "B200_PCIe-PyTorch-1, B200_PCIe-PyTorch-2, B200_PCIe-PyTorch-3" |
|
PR_Github #15665 [ run ] triggered by Bot |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/integration/test_lists/waives.txt (1)
293-297: Deduplicate SKIP entry for TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_tp8]The same test selector appears twice in tests/integration/test_lists/waives.txt (lines 293 and 296), which can confuse maintainers and any tooling that parses these waives. Please consolidate into a single entry with both bug references. For example, remove the second occurrence (line 296) and update the first to include both NVBug URLs:
-accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_tp8] SKIP (https://nvbugs/5445466) +accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[throughput_tp8] SKIP (https://nvbugs/5442827,https://nvbugs/5445466)This keeps a single, authoritative waive for that test.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
tests/integration/defs/accuracy/test_llm_api_pytorch.py(1 hunks)tests/integration/test_lists/waives.txt(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/integration/defs/accuracy/test_llm_api_pytorch.py
|
PR_Github #15665 [ run ] completed with state |
Signed-off-by: bhsueh <[email protected]>
Signed-off-by: bhsueh <[email protected]>
Signed-off-by: bhsueh <[email protected]>
|
/bot run --stage-list "B200_PCIe-PyTorch-1, B200_PCIe-PyTorch-2, B200_PCIe-PyTorch-3" |
There was a problem hiding this comment.
Actionable comments posted: 0> [!CAUTION]
Some comments are outside the diff and can’t be posted inline due to platform limits.
🔭 Outside diff range comments (1)
tests/integration/test_lists/waives.txt (1)
295-313: Ensure waived tests exist in the treeThe current verification script hard-codes file paths that don’t match our repo layout, so none of the waived entries can be found. To catch typos early, update it to search across all Python files:
File: tests/integration/test_lists/waives.txt (lines 295–313)
Suggested replacement verification script:
#!/usr/bin/env bash set -euo pipefail fail=0 check_class_method() { local class="$1" method="$2" if ! rg -nPzo "(?s)class\s+${class}\b.*?def\s+${method}\s*\(" --glob '*.py' > /dev/null; then echo "MISS: ${class}::${method}" fail=1 else echo "OK: ${class}::${method}" fi } check_function() { local func="$1" if ! rg -nP "def\s+${func}\s*\(" --glob '*.py' > /dev/null; then echo "MISS: ${func}" fail=1 else echo "OK: ${func}" fi } # Class-based tests check_class_method TestMistralSmall24B test_auto_dtype check_class_method TestMistral_Nemo_12B_Base test_fp8 check_class_method TestLlama3_1_8BInstruct test_ctx_pp_gen_tp_asymmetric check_class_method TestQwen3_8B test_nixl_backend check_class_method TestDeepSeekV3Lite test_nixl_backend check_class_method TestDeepSeekR1 test_nvfp4_multi_gpus # Function-based tests check_function test_llm_api_single_gpu_with_mpirun check_function test_gpt_ib check_function test_gpt_ib_streaming check_function test_gpt_ib_ptuning check_function test_mistral_ib_mm check_function test_t5_ib check_function test_gpt_speculative_decoding_bls check_function test_ptp_quickstart_multimodal check_function test_llmapi_speculative_decoding_mtp check_function test_disaggregated_genbs1 if (( fail > 0 )); then echo echo "ERROR: One or more waived tests could not be found. Please verify node IDs." exit 1 fi echo echo "✅ All waived tests resolved."• Replace your existing script with the above (or equivalent) to avoid hard-coded paths
• This will scan all*.pyfiles for each class or function, catching typos without assumptions about directory structure
🧹 Nitpick comments (1)
tests/integration/test_lists/waives.txt (1)
295-313: Nit: Standardize NVBug links for consistency and tooling.This file mixes https://nvbugs/, https://nvbugspro.nvidia.com/bug/, and others. For consistency and to ease linkification in tools, prefer one canonical form (commonly nvbugspro.nvidia.com/bug/) for new entries.
Apply this diff to normalize the newly added entries in this hunk:
-accuracy/test_llm_api_pytorch.py::TestMistralSmall24B::test_auto_dtype SKIP (https://nvbugs/5454875) +accuracy/test_llm_api_pytorch.py::TestMistralSmall24B::test_auto_dtype SKIP (https://nvbugspro.nvidia.com/bug/5454875) -examples/test_llm_api_with_mpi.py::test_llm_api_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/5434372) +examples/test_llm_api_with_mpi.py::test_llm_api_single_gpu_with_mpirun[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugspro.nvidia.com/bug/5434372) -triton_server/test_triton.py::test_gpt_ib[gpt-ib] SKIP (https://nvbugs/5431116) +triton_server/test_triton.py::test_gpt_ib[gpt-ib] SKIP (https://nvbugspro.nvidia.com/bug/5431116) -disaggregated/test_workers.py::test_workers_kv_cache_events[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/5457504) +disaggregated/test_workers.py::test_workers_kv_cache_events[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugspro.nvidia.com/bug/5457504) -accuracy/test_llm_api.py::TestMistral_Nemo_12B_Base::test_fp8 SKIP (https://nvbugs/5413197) +accuracy/test_llm_api.py::TestMistral_Nemo_12B_Base::test_fp8 SKIP (https://nvbugspro.nvidia.com/bug/5413197) -triton_server/test_triton.py::test_gpt_ib_streaming[gpt-ib-streaming] SKIP (https://nvbugs/5371349) +triton_server/test_triton.py::test_gpt_ib_streaming[gpt-ib-streaming] SKIP (https://nvbugspro.nvidia.com/bug/5371349) -triton_server/test_triton.py::test_gpt_ib_ptuning[gpt-ib-ptuning] SKIP (https://nvbugs/5445624) +triton_server/test_triton.py::test_gpt_ib_ptuning[gpt-ib-ptuning] SKIP (https://nvbugspro.nvidia.com/bug/5445624) -triton_server/test_triton.py::test_mistral_ib_mm[mistral-ib-mm] SKIP (https://nvbugs/5371343) +triton_server/test_triton.py::test_mistral_ib_mm[mistral-ib-mm] SKIP (https://nvbugspro.nvidia.com/bug/5371343) -triton_server/test_triton.py::test_t5_ib[t5-ib] SKIP (https://nvbugs/5456482) +triton_server/test_triton.py::test_t5_ib[t5-ib] SKIP (https://nvbugspro.nvidia.com/bug/5456482) -triton_server/test_triton_llm.py::test_gpt_speculative_decoding_bls[False-False-1---False-True-True-0-128-disableDecoupleMode-inflight_fused_batching-disableTrtOverlap-0.2-guaranteed_no_evict---1-1-1-False-ensemble] SKIP (https://nvbugs/5456485) +triton_server/test_triton_llm.py::test_gpt_speculative_decoding_bls[False-False-1---False-True-True-0-128-disableDecoupleMode-inflight_fused_batching-disableTrtOverlap-0.2-guaranteed_no_evict---1-1-1-False-ensemble] SKIP (https://nvbugspro.nvidia.com/bug/5456485) -accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=4] SKIP (https://nvbugs/5434320) +accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric[GSM8K-gen_tp=1-ctx_pp=4] SKIP (https://nvbugspro.nvidia.com/bug/5434320) -accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_nixl_backend SKIP (https://nvbugs/5448437) +accuracy/test_disaggregated_serving.py::TestQwen3_8B::test_nixl_backend SKIP (https://nvbugspro.nvidia.com/bug/5448437) -accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_nixl_backend SKIP (https://nvbugs/5448437) +accuracy/test_disaggregated_serving.py::TestDeepSeekV3Lite::test_nixl_backend SKIP (https://nvbugspro.nvidia.com/bug/5448437) -accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency_trtllmgen] SKIP (https://nvbugs/5445466) +accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency_trtllmgen] SKIP (https://nvbugspro.nvidia.com/bug/5445466) -accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency] SKIP (https://nvbugs/5445466) +accuracy/test_llm_api_pytorch.py::TestDeepSeekR1::test_nvfp4_multi_gpus[latency] SKIP (https://nvbugspro.nvidia.com/bug/5445466) -test_e2e.py::test_ptp_quickstart_multimodal[mistral-small-3.1-24b-instruct-Mistral-Small-3.1-24B-Instruct-2503-image-True] SKIP (https://nvbugs/5459817) +test_e2e.py::test_ptp_quickstart_multimodal[mistral-small-3.1-24b-instruct-Mistral-Small-3.1-24B-Instruct-2503-image-True] SKIP (https://nvbugspro.nvidia.com/bug/5459817) -llmapi/test_llm_examples.py::test_llmapi_speculative_decoding_mtp SKIP (https://nvbugs/5461796) +llmapi/test_llm_examples.py::test_llmapi_speculative_decoding_mtp SKIP (https://nvbugspro.nvidia.com/bug/5461796) -disaggregated/test_disaggregated.py::test_disaggregated_genbs1[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugs/5459811) +disaggregated/test_disaggregated.py::test_disaggregated_genbs1[TinyLlama-1.1B-Chat-v1.0] SKIP (https://nvbugspro.nvidia.com/bug/5459811)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
tests/integration/defs/accuracy/test_llm_api_pytorch.py(1 hunks)tests/integration/test_lists/waives.txt(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/integration/defs/accuracy/test_llm_api_pytorch.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (1)
tests/integration/test_lists/waives.txt (1)
295-313: LGTM on newly added waives; reasons are documented with NVBug IDs.The new SKIPs are scoped and justified, aligning with the PR’s goal to unwaive some Qwen tests while quarantining unrelated instabilities. No functional code impact here.
|
PR_Github #15712 [ run ] triggered by Bot |
|
PR_Github #15712 [ run ] completed with state |
|
/bot run --stage-list "B200_PCIe-PyTorch-1" --debug |
|
PR_Github #15755 [ run ] triggered by Bot |
|
PR_Github #15755 [ run ] completed with state |
|
/bot run --debug --stage-list "B200_PCIe-PyTorch-1" |
|
PR_Github #15805 [ run ] triggered by Bot |
|
PR_Github #15805 [ run ] completed with state |
Signed-off-by: bhsueh <[email protected]>
Signed-off-by: bhsueh <[email protected]>
|
/bot run --stage-list "B200_PCIe-PyTorch-1" |
|
PR_Github #15869 [ run ] triggered by Bot |
|
PR_Github #15869 [ run ] completed with state |
|
/bot run --stage-list "B200_PCIe-PyTorch-1" |
|
PR_Github #15926 [ run ] triggered by Bot |
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
tests/integration/defs/accuracy/test_llm_api_pytorch.py(1 hunks)tests/integration/test_lists/waives.txt(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- tests/integration/defs/accuracy/test_llm_api_pytorch.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
|
PR_Github #15926 [ run ] completed with state |
|
/bot skip --comment "unwaive CI tests are tested" |
|
PR_Github #15962 [ skip ] triggered by Bot |
|
PR_Github #15962 [ skip ] completed with state |
Signed-off-by: bhsueh <[email protected]> Signed-off-by: Yuxin <[email protected]>
Summary by CodeRabbit