Repository navigation
[https://nvbugs/7113609][infra] Waive 1 failed case(s) on CPU - #22543
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe CPU test waiver list now skips ChangesCPU Test Waiver
Estimated code review effort: 1 (Trivial) | ~2 minutes Change: Other Merge Risk:
|
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
trtllm-agent-review
left a comment
There was a problem hiding this comment.
The waive itself is correct. The id resolves, and the full:CPU/ prefix matches the CPU-Generic x86 and arm stages, which are the only stages that run this file (l0_cpu.yml:146). The case fails deterministically on f1856a4134 and on current main b5692a546f, with the signature in the description, and the line cites the bug filed for it. Posted 2 comments: 0 Blocker, 0 Possible blocker, 2 Nice to have. Both are about coordinating with the sibling waive PRs for the same file and with the open fix #22545.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) This is one of three cases of test_modeling_v2_k3_fp4_mla_copies.py that fail on CPU for the same reason: #21996 copied the FP4 MLA backend definitions from before #22078 changed them. The other two are waived in separate open PRs that insert at this same line: #22536 (test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels], nvbugs/7113412) and #22544 (test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache], nvbugs/7113610). Whichever lands second will conflict here. Until all three are in, the file's l0_cpu entry stays red on CPU-Generic x86/arm, because all three cases fail on f1856a4134 and on main b5692a546f. Suggest landing the three lines together (one PR, or back to back with a rebase) and marking the three bugs as duplicates, so one fix closes them all.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) For nvbugs/7113609: the open fix #22545 makes this case pass; with its changes to the K3 copies and to this test, all 8 cases of the file pass. It does not touch waives.txt or reference this bug, though. If this waive lands first, please have #22545 or a follow-up delete this line, plus the sibling lines from #22536/#22544, and link the bugs. Otherwise the parity check of all 33 copied definitions in _k3_fp4_mla_cache.py stays skipped after the fix. If #22545 lands first, this PR can just be closed. Also, the case stops at its first mismatch, so the bug's scope is wider than its title: 13 of the 33 definitions no longer match a backend source, not only can_fuse_fp4_mla_q_quant.
|
Closing: nvbugs/7113609 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 1 failed case(s) on
CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]broken on main for CPU: failure signatureassertionerror: _k#_fp#_mla_cache.can_fuse_fp#_mla...fers from its source in [cache_update] assert [], onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113609 tracks the fix. ScopeCPUis the stage prefix that ran the failing test.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]on CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
CPUmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.07ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-2563b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]: incident4234c53ee1a8054e7858cc0d6e6f9d4fb24cb027dbf7121bdb18a3a9b8cf604c, observation309e44f19e4e1a26054a4c3d1977d265cf7751f89ec0821c0e54fc2a1ef36094Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txtgains one line at line 70. The line adds a CPU skip entry fortest_copied_definitions_match_their_sources[_k3_fp4_mla_cache], tracked by nvbugs/7113609.mainin copied definitions that differ from their source incache_update. The PR does not establish thatmainis fixed. The waiver stays in place until the bug is fixed or closed and the test passes on CPU post-merge.QA Engineer Review
Test-list files changed:
tests/integration/test_lists/waives.txt(+1/-0).full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609).test-db/orqa/lists changed.mainfailure incache_updatestays unverified by CI until the waiver is removed.needs follow-up. Remove the waiver after the bug is fixed and the test passes on CPU.Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds a CPU skip fortest_copied_definitions_match_their_sources[_k3_fp4_mla_cache]. QA should confirm that the test passes on CPU once nvbugs/7113609 is resolved, then remove this entry.