Repository navigation
[https://nvbugs/7034158][test] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300 - #22260
Conversation
…test on GB300 The warmup illegal memory access is fixed by the MNNVL fused RMSNorm shared-memory allocation fix (NVIDIA#21785). Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
|
@coderabbitai review |
Action performed
|
|
No actionable comments were generated in the recent review. WalkthroughThe integration test configuration sets four-node launches for three Qwen3 accuracy cases. The waive list removes the skip entry for the GB300 ChangesQwen3 Test Configuration
Estimated code review effort: 2 (Simple) | ~10 minutes Change: Other Merge Risk:
|
|
/bot run |
|
PR_Github #84791 [ run ] triggered by Bot. Commit: |
|
PR_Github #84791 [ run ] completed with state |
sunnyqgg
left a comment
There was a problem hiding this comment.
The fix this unwaive depends on, #21785 (b38f901), is already on the base. It removes kernel_mt's fixed smem=1024 + tpc*hidden*2 reservation, which is exactly the 230400-vs-231200-byte overrun nvbug 7034158 reports, and no other waives.txt entry still skips this test. The pre-merge run on 308bb47 (L0_MergeRequest_PR #70155) passed, though it does not run this test. Counts: 0 blockers, 0 possible blockers, 2 required, 1 nice to have.
- [Required]
tests/integration/test_lists/qa/llm_function_multinode.yml:43: the unwaived case has nolaunchof its own, so it inheritsdefaults.launch(lines 4-8:mode: mpi,nodes: 2,world_size: 16).qa/README.md:65-71says an MPI case runsworld_sizeranks on exactlynodeshosts, and launch plans that cannot meet both are rejected. GB300 hosts have 4 GPUs each (GB300-16_GPUs-4_Nodesinjenkins/L0_Test.groovy:7665, andtest-db/l0_gb300_multi_nodes_node4_gpu16.yml:5). The verification in the description ran 16 GPUs across 4 nodes. Under that launch contract, the GB300 QA job cannot place 16 ranks on 2 hosts, so the GB300 run this PR re-enables is rejected at launch planning and nvbug 7034158 closes with no QA run of the test. Please give this entry its own launch withnodes: 4(modeandworld_sizestay inherited), or state in the PR how the GB300 QA job places it. - [Required] PR title: this PR only edits
waives.txt; the code fix landed in #21785. CONTRIBUTING.md asks for Conventional Commits titles, wherefixmarks a change that patches code, and unwaive-only PRs on main use[test](e.g. #22215, #22087, #22126). Please retitle to[https://nvbugs/7034158][test] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300.
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
These cases need 16 ranks in one NVL72 on 4-GPU Grace-Blackwell hosts; the default requests two nodes, which cannot accommodate 16 ranks on these hosts. Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
|
/bot skip --comment "Test-list-only change: removes a GB300 QA waive and sets nodes: 4 on three QA multinode YAML entries; none of the touched tests run in pre-merge. Full pipeline #70155 passed on the waive change; Pre-commit (check_test_list --validate) passes on bb1640c." |
|
PR_Github #84846 [ skip ] triggered by Bot. Commit: |
|
PR_Github #84846 [ skip ] completed with state |
… QA cases The QA launcher derives MPI node count from per-host GPU capacity, so the nodes: 4 overrides have no effect; keep the defaults. Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
|
/bot skip --comment "Waive-list-only change; the unwaived test is not in pre-merge." |
|
PR_Github #85419 [ skip ] triggered by Bot. Commit: |
|
PR_Github #85419 [ skip ] completed with state |
Dev Engineer Review
tests/integration/test_lists/waives.txt: removes one line, the GB300 SKIP forTestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm(nvbugs/7034158). The test runs on GB300 after this change.tests/integration/test_lists/qa/llm_function_multinode.yml: addslaunch: nodes: 4to three cases:test_fp8_adp16_deepgemm(no launch block before)test_fp8_tp16_mtp3_trtllm(no launch block before)test_nvfp4_adp16_cutedsl(no launch block before)test_fp8_tp16_mtp3_trtllmcase. Confirm that thetest_fp8_adp16_deepgemmandtest_nvfp4_adp16_cutedslnode changes are intended.#21785. This diff contains no source change. The fix must already exist onmain. Confirm this before merge. Without it, the test fails during warmup with the illegal memory access.QA Engineer Review
tests/integration/test_lists/waives.txt(1 deletion).tests/integration/test_lists/qa/llm_function_multinode.yml(6 additions).full:GB300/accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm, previouslySKIP (https://nvbugs/7034158).TestQwen3_8_2_4T_A95Bcases in the manual QA list now run on 4 nodes.qa/llm_function_multinode.yml. It is not in the pre-mergetest-db/list. This diff does not change either list.needs follow-up. Confirm the GB300 run passes in CI. Confirm the 4-node launch works for all three updated cases.Per-File QA Perspective
tests/integration/test_lists/waives.txt: Removes the GB300 SKIP forTestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm. QA should confirm the test passes on GB300 before the waiver stays removed.tests/integration/test_lists/qa/llm_function_multinode.yml: Setsnodes: 4for three 16-rank Qwen3 cases. QA should confirm each case launches and completes on four nodes.Description
Removes the GB300 waive for
accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm(NVBug 7034158). The test crashed with an illegal memory access during warmup because the MNNVL fused allreduce + RMSNorm kernel reserved too little shared memory (230400 bytes allocated, 231200 used). #21785 fixed this; it is the same root cause as NVBug 7028063.Test Coverage
accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllmis in the QA listqa/llm_function_multinode.ymlonly, not in the pre-merge test-db, so this PR's CI does not run it. It was run by hand on GB300, 16 ranks on 4 nodes x 4 GPUs in one NVL72:mnnvl_fused_allreduce.pyandmnnvl_allreduce_norm.pyexactly as merged in [https://nvbugs/7028063][fix] Correct MNNVL fused RMSNorm shared-memory allocation #21785 (both unchanged on main since): passes, with no shared-memory overrun message.PR Checklist
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.