Repository navigation
[https://nvbugs/7113610][infra] Waive 1 failed case(s) on CPU - #22544
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe integration test waiver list adds a CPU skip entry for the Kimi K3 FP4 MLA cache backend-contract import test, referencing NVBUG 7113610. ChangesCPU test waiver
Estimated code review effort: 1 (Trivial) | ~3 minutes Change: Other Merge Risk:
|
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
/bot skip --comment "auto-waive: waive-only change" |
|
PR_Github #85730 [ skip ] triggered by Bot. Commit: |
trtllm-agent-review
left a comment
There was a problem hiding this comment.
I checked the one-line CPU waive. The entry is well formed and sorted, and its full:CPU/ prefix matches every CPU-Generic x86/arm stage that runs l0_cpu.yml. It skips only the failing parametrization (the other three still run and pass), and it cites the bug filed for this failure. Posted comments: 0 blockers, 0 possible blockers, 2 nice to have. Both are about merge coordination: the two sibling waive PRs for the same file, and the in-flight fix #22545.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610) |
There was a problem hiding this comment.
[Nice to have] (R2: Test lists and waives) For nvbugs/7113610, the cause and the fix are already known. #21996 made _k3_fp4_mla_cache.py:60 and kernels/k3r_fp4_mla/fp4_mla_kv_append.py:46 import _fp4_mla_generation_num_blocks_device, which #22078 had removed from fp4_mla/metadata.py. The open #22545 defines that helper in _k3_fp4_mla_cache.py. With its diff applied to this base, all 8 cases of test_modeling_v2_k3_fp4_mla_copies.py pass, this one included.
#22545 does not touch waives.txt, though, so this line and the fix race. If #22545 merges first, please close this PR rather than merge a waive for a passing test. Otherwise, please have #22545 (or the bug owner) delete this line, and the sibling lines from #22543/#22536, in the same change. Please also link #22545 in the bug, which is still unprioritized.
| full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440) | ||
| full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626) | ||
| full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) This is one of three open auto-waive PRs for the same file and the same root cause. #22543 (nvbugs/7113609) waives test_copied_definitions_match_their_sources[_k3_fp4_mla_cache], and #22536 (nvbugs/7113412) waives test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]. Running the file on this tree gives exactly these three failures (3 failed, 5 passed), all from #21996 landing after #22078 changed the FP4 MLA backend.
Merged alone, this PR does not turn the CPU stages green. The PRs listed in the description still fail on the other two cases, and so does this PR's own CPU run, because change-based selection runs the whole file. All three PRs also insert after line 69, so whichever merges second conflicts at this line: git merge-file of this PR with #22543 conflicts here.
Please land the three together, either folded into one PR or merged back to back with a rebase. Please also mark nvbugs/7113609, 7113412 and 7113610 as one root cause, so a single fix closes all three.
|
PR_Github #85730 [ skip ] completed with state |
|
Closing: nvbugs/7113610 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 1 failed case(s) on
CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]broken on main for CPU: failure signatureassertionerror: tensorrt_llm._torch.attention.back...evice = <ast.alias object at #x#>.name, onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113610 tracks the fix. ScopeCPUis the stage prefix that ran the failing test.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]on CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
CPUmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.07ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-2563b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]: incident2481ff971662a7cec996889b7f819a4875af3a8d682e3251829daad8115ccc40, observationfd5ac07ee9742bcbf5c194b1a9eb03a953bf9a097cea2ee25bb0f26c47115868Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt. No source code, configuration, or test code changes.test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]on the CPU stage. It links to NVBUG 7113610.CPU/...). Confirm the full selector matches the failing test ID exactly. A mismatch leaves the test running and failing.QA Engineer Review
tests/integration/test_lists/waives.txt, +1 line.full:CPU/...::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610).unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache].test-db/orqa/lists.Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds a CPU skip for one KIMI K3 FP4 MLA copies import-contract test. QA should confirm the test passes on CPU once the bug is resolved and the waiver is removed.