Skip to content

[https://nvbugs/7034158][test] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300 - #22260

Merged
pranav-nvidia merged 3 commits into
NVIDIA:mainfrom
pranav-nvidia:fix/unwaive-nvbug-7034158
Nov 17, 2026
Merged

pranav-nvidia merged 3 commits into
NVIDIA:mainfrom
pranav-nvidia:fix/unwaive-nvbug-7034158

Conversation

@pranav-nvidia

@pranav-nvidia pranav-nvidia commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Dev Engineer Review

  • tests/integration/test_lists/waives.txt: removes one line, the GB300 SKIP for TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm (nvbugs/7034158). The test runs on GB300 after this change.
  • tests/integration/test_lists/qa/llm_function_multinode.yml: adds launch: nodes: 4 to three cases:
    • test_fp8_adp16_deepgemm (no launch block before)
    • test_fp8_tp16_mtp3_trtllm (no launch block before)
    • test_nvfp4_adp16_cutedsl (no launch block before)
  • The commit message gives the reason for the node count. The default requests two nodes, which cannot host 16 ranks on the 4-GPU hosts. The objectives mention only the test_fp8_tp16_mtp3_trtllm case. Confirm that the test_fp8_adp16_deepgemm and test_nvfp4_adp16_cutedsl node changes are intended.
  • The PR objectives attribute the kernel fix to #21785. This diff contains no source change. The fix must already exist on main. Confirm this before merge. Without it, the test fails during warmup with the illegal memory access.
  • No source code or runtime configuration changed.

QA Engineer Review

  • Modified: tests/integration/test_lists/waives.txt (1 deletion).
  • Modified: tests/integration/test_lists/qa/llm_function_multinode.yml (6 additions).
  • Removed waiver: full:GB300/accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm, previously SKIP (https://nvbugs/7034158).
  • Launch change: three TestQwen3_8_2_4T_A95B cases in the manual QA list now run on 4 nodes.
  • Covered behavior: the MNNVL fused allreduce + RMSNorm path with 16 GPUs across four nodes. The author reports a GSM8K score of 93.556 against a reference of 93.800 on GB300, with the fix present.
  • Test list placement: the test is in qa/llm_function_multinode.yml. It is not in the pre-merge test-db/ list. This diff does not change either list.
  • CI results on the current head for the GB300 run: not available from this diff.
  • Coverage verdict: needs follow-up. Confirm the GB300 run passes in CI. Confirm the 4-node launch works for all three updated cases.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Removes the GB300 SKIP for TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm. QA should confirm the test passes on GB300 before the waiver stays removed.
  • tests/integration/test_lists/qa/llm_function_multinode.yml: Sets nodes: 4 for three 16-rank Qwen3 cases. QA should confirm each case launches and completes on four nodes.

Description

Removes the GB300 waive for accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm (NVBug 7034158). The test crashed with an illegal memory access during warmup because the MNNVL fused allreduce + RMSNorm kernel reserved too little shared memory (230400 bytes allocated, 231200 used). #21785 fixed this; it is the same root cause as NVBug 7028063.

Test Coverage

accuracy/test_llm_api_pytorch.py::TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm is in the QA list qa/llm_function_multinode.yml only, not in the pre-merge test-db, so this PR's CI does not run it. It was run by hand on GB300, 16 ranks on 4 nodes x 4 GPUs in one NVL72:

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

…test on GB300

The warmup illegal memory access is fixed by the MNNVL fused RMSNorm shared-memory allocation fix (NVIDIA#21785).

Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
@pranav-nvidia

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration test configuration sets four-node launches for three Qwen3 accuracy cases. The waive list removes the skip entry for the GB300 TestQwen3_8_2_4T_A95B::test_fp8_tp16_mtp3_trtllm test.

Changes

Qwen3 Test Configuration

Layer / File(s) Summary
Set per-case node counts
tests/integration/test_lists/qa/llm_function_multinode.yml
The test_fp8_adp16_deepgemm, test_fp8_tp16_mtp3_trtllm, and test_nvfp4_adp16_cutedsl cases now set launch.nodes to 4.
Remove the test skip entry
tests/integration/test_lists/waives.txt
The waive list no longer contains the skip entry for the GB300 Qwen3 test associated with nvbugs/7034158.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Merge Risk: Minimal · up to bb164

This PR re-enables a previously waived GB300 Qwen3 accuracy test and sets four-node launches for three 16-rank QA cases. It affects only test configuration, not product behavior. No concrete defect was found, and the author reports a passing GB300 run with the kernel fix in place.

Security architecture risk: Minimal · up to bb164

The inspected changes request four nodes for three existing accuracy tests and remove one GB300 skip. No introduced or worsened security, isolation, ownership, or rollback issue was established. The external test runner’s allocation and credential handling could not be verified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The directly supported scope is scheduling configuration for three existing QA cases and GB300 eligibility for one case. These edits alone do not establish cross-tenant access, elevated privileges, or production exposure.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the NVBug ID and type tag in the required format and states the main change: removing the GB300 waiver for the Qwen3 FP8 TP16 MTP3 accuracy test. It matches the waives.txt and multinod…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The PR removes the GB300 waive for the Qwen3 8.2 4T A95B FP8 TP16 MTP3 TRT-LLM accuracy test (NVBug 7034158). The description explains the problem (illegal memory access during warmup from insufficien…
Full details: Title check

Explanation

The title names the NVBug ID and type tag in the required format and states the main change: removing the GB300 waiver for the Qwen3 FP8 TP16 MTP3 accuracy test. It matches the waives.txt and multinode QA config changes.

Full details: Description check

Explanation

The PR removes the GB300 waive for the Qwen3 8.2 4T A95B FP8 TP16 MTP3 TRT-LLM accuracy test (NVBug 7034158). The description explains the problem (illegal memory access during warmup from insufficient shared memory in the MNNVL fused allreduce + RMSNorm kernel), states that #21785 fixed it, and gives manual GB300 test results showing a pass. It also explains why CI does not run the test. The Description and Test Coverage sections are filled in with specific information. The PR Checklist is only ticked as a single item and the other template items are not addressed, which is a minor gap. The description is also not in the exact title or template format, and the template's Description section is not labelled as such in the body, but the required information is present.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 308bb47 and bb1640c.

Files selected for processing: 1

  • tests/integration/test_lists/qa/llm_function_multinode.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@pranav-nvidia
pranav-nvidia marked this pull request as ready for review November 13, 2026 17:15
@pranav-nvidia

Copy link
Copy Markdown
Contributor Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84791 [ run ] triggered by Bot. Commit: 308bb47 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84791 [ run ] completed with state SUCCESS. Commit: 308bb47
/LLM/main/L0_MergeRequest_PR pipeline #70155 completed with status: 'SUCCESS'

CI Report

Link to invocation

@sunnyqgg sunnyqgg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix this unwaive depends on, #21785 (b38f901), is already on the base. It removes kernel_mt's fixed smem=1024 + tpc*hidden*2 reservation, which is exactly the 230400-vs-231200-byte overrun nvbug 7034158 reports, and no other waives.txt entry still skips this test. The pre-merge run on 308bb47 (L0_MergeRequest_PR #70155) passed, though it does not run this test. Counts: 0 blockers, 0 possible blockers, 2 required, 1 nice to have.

  • [Required] tests/integration/test_lists/qa/llm_function_multinode.yml:43: the unwaived case has no launch of its own, so it inherits defaults.launch (lines 4-8: mode: mpi, nodes: 2, world_size: 16). qa/README.md:65-71 says an MPI case runs world_size ranks on exactly nodes hosts, and launch plans that cannot meet both are rejected. GB300 hosts have 4 GPUs each (GB300-16_GPUs-4_Nodes in jenkins/L0_Test.groovy:7665, and test-db/l0_gb300_multi_nodes_node4_gpu16.yml:5). The verification in the description ran 16 GPUs across 4 nodes. Under that launch contract, the GB300 QA job cannot place 16 ranks on 2 hosts, so the GB300 run this PR re-enables is rejected at launch planning and nvbug 7034158 closes with no QA run of the test. Please give this entry its own launch with nodes: 4 (mode and world_size stay inherited), or state in the PR how the GB300 QA job places it.
  • [Required] PR title: this PR only edits waives.txt; the code fix landed in #21785. CONTRIBUTING.md asks for Conventional Commits titles, where fix marks a change that patches code, and unwaive-only PRs on main use [test] (e.g. #22215, #22087, #22126). Please retitle to [https://nvbugs/7034158][test] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300.

Comment thread tests/integration/test_lists/waives.txt Resolved
@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@pranav-nvidia pranav-nvidia changed the title [https://nvbugs/7034158][fix] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300 [https://nvbugs/7034158][test] Unwaive Qwen3.8 FP8 TP16 MTP3 accuracy test on GB300 Nov 13, 2026
These cases need 16 ranks in one NVL72 on 4-GPU Grace-Blackwell hosts; the default requests two nodes, which cannot accommodate 16 ranks on these hosts.

Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
@pranav-nvidia
pranav-nvidia requested a review from a team as a code owner November 14, 2026 00:05
@pranav-nvidia

Copy link
Copy Markdown
Contributor Author

/bot skip --comment "Test-list-only change: removes a GB300 QA waive and sets nodes: 4 on three QA multinode YAML entries; none of the touched tests run in pre-merge. Full pipeline #70155 passed on the waive change; Pre-commit (check_test_list --validate) passes on bb1640c."

@pranav-nvidia
pranav-nvidia enabled auto-merge (squash) November 14, 2026 00:40
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84846 [ skip ] triggered by Bot. Commit: bb1640c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84846 [ skip ] completed with state SUCCESS. Commit: bb1640c
Skipping testing for commit bb1640c

Link to invocation

@LarryXFly
LarryXFly requested a review from xinhe-nv November 14, 2026 02:12
… QA cases

The QA launcher derives MPI node count from per-host GPU capacity, so the
nodes: 4 overrides have no effect; keep the defaults.

Signed-off-by: Pranav Shrestha <254760092+pranav-nvidia@users.noreply.github.com>
@pranav-nvidia

Copy link
Copy Markdown
Contributor Author

/bot skip --comment "Waive-list-only change; the unwaived test is not in pre-merge."

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85419 [ skip ] triggered by Bot. Commit: 6a3f946 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85419 [ skip ] completed with state SUCCESS. Commit: 6a3f946
Skipping testing for commit 6a3f946

Link to invocation

@pranav-nvidia
pranav-nvidia merged commit c2a957c into NVIDIA:main Nov 17, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants