Repository navigation
[https://nvbugs/7113412][infra] Waive 1 failed case(s) on CPU - #22536
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe CPU skip list now includes ChangesCPU skip-list update
Estimated code review effort: 1 (Trivial) | ~5 minutes Change: Other Merge Risk:
|
trtllm-agent-review
left a comment
There was a problem hiding this comment.
I checked this one-line waive against the CI results of the runs listed in the description. The line is well formed and applies to every CPU-* stage, x86 and arm. It skips only a case that fails on main and cites the bug filed for it. Comments: 0 blockers, 0 possible blockers, 3 nice to have. They cover coordination with the sibling waive PRs #22543 and #22544, a merge conflict with current main, and the open fix #22545.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412) |
There was a problem hiding this comment.
[Nice to have] (R2: Test lists and waives) This line waives 1 of the 3 cases of test_modeling_v2_k3_fp4_mla_copies.py that fail on main from the same root cause: #21996 copied the FP4 MLA backend as it was before #22078 removed _fp4_mla_attention_v_repack_kernel and _fp4_mla_generation_num_blocks_device. The other two cases fail in every run behind the PRs listed in the description (e.g. L0_MergeRequest_PR 70898, CPU-Generic-x86-1 and CPU-Generic-arm-1):
test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]
They are waived separately, in #22543 (nvbugs/7113609) and #22544 (nvbugs/7113610). l0_cpu.yml lists this file as a whole, so this PR on its own leaves both CPU stages red, and the six PRs listed above still fail after a rebase. Suggest putting the three lines in one PR under one bug (marking 7113609/7113610 as duplicates of 7113412), or merging the three PRs together.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412) |
There was a problem hiding this comment.
[Nice to have] (R2: Test lists and waives) For nvbugs/7113412, which has no engineer assigned yet: #22545, from the author of #21996, already fixes all three failing cases of this file and keeps their test ids. It does not reference this bug or 7113609/7113610, though, so nothing ties the fix to removing this line. Please assign the bug and link it from #22545:
- If [None][fix] Kimi K3 Modeling V2: follow the FP4 MLA backend API on main #22545 merges first, this PR can be closed.
- If this PR merges first, [None][fix] Kimi K3 Modeling V2: follow the FP4 MLA backend API on main #22545 should delete this line and the sibling ones.
Otherwise [_k3_fp4_mla_kernels] stays skipped after it passes again, and nothing on CPU checks the other 23 copied definitions in _k3_fp4_mla_kernels.py against the backend. FYI, the waived case has four orphaned definitions, not only the one in the failure message: the three _fp4_mla_attention_v_repack* kernels and fp4_mla_repack_v_cache.
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412) | ||
| full:CPU/unittest/_torch/modeling_v2/test_modeling_v2_ci_selection.py::test_every_test_is_in_a_test_list SKIP (https://nvbugs/7082290) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) This branch no longer merges into main (GitHub reports conflicts). #22510 (ceb406c) deleted this test_modeling_v2_ci_selection.py::test_every_test_is_in_a_test_list ... (https://nvbugs/7082290) line, right below the new waive. Please regenerate the branch on current main. #22543 and #22544 insert at the same position, so after any one of the three merges, the other two conflict again. One PR carrying all three lines avoids that.
|
Closing: nvbugs/7113412 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 1 failed case(s) on
CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]broken on main for CPU: failure signatureassertionerror: _k#_fp#_mla_kernels._fp#_mla_atten...on, <ast.functiondef object at #x#>)]}, onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113412 tracks the fix. ScopeCPUis the stage prefix that ran the failing test.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]on CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
CPUmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.0b63a992e-b7c5-4815-92d3-fe7b0737fe16; snapshot SHA-2565fb2f3fdddfe9ce39251cec27d66dc9e9c21975354c430fd53248a3ea4b859d8; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]: incident41da36aea844a2574e27179e79f8e7546e2a01f1fb0600a093ac95184ae2920a, observation59881bcfdc1c01b507ada7ff7043613fac263c675bea789b372f41b7e6a477c0Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt. It adds a CPU skip forunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels], linked to NVBUG 7113412.mainfrom 2026-11-18T02:11:36Z to 2026-11-18T04:58:01Z. The PR does not establish thatmainis fixed.QA Engineer Review
tests/integration/test_lists/waives.txt(+1 line, 0 removed).test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]on CPU, with reasonhttps://nvbugs/7113412.needs follow-up. QA should confirm the failure onmainis resolved and remove the waiver when the bug is closed.Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds a CPU skip for one kimi_k3_vl FP4 MLA copy-check case. QA should verify that the case passes on CPU once the bug is closed, then remove this entry.