Repository navigation
[https://nvbugs/7113273][infra] Waive 1 failed case(s) on all platforms - #22524
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe integration test skip list adds ChangesSkip-list update
Estimated code review effort: 1 (Trivial) | ~2 minutes Change: Other Merge Risk:
|
trtllm-agent-review
left a comment
There was a problem hiding this comment.
No blockers. Posted 3 comments, all Nice to have: 2 inline on tests/integration/test_lists/waives.txt:171 and 1 below. The waive itself is correct. The test fails on main and on the merge base with the ImportError described below (CPU-Generic-x86-3/arm-3 in L0_MergeRequest_PR 70869, 70902 and 70908), the entry matches how apply_waives_ut applies unittest waives, and it cites the bug filed for this failure.
- [Nice to have] (N3: Missing or ineffective tests) For nvbugs/7113273 (no owner, Unprioritized): this waive removes the only CPU signal for a break that is not test-only. #22078 removed
_fp4_mla_generation_num_blocks_devicefromfp4_mla/metadata.pyand madeshareda required keyword-only argument of_ensure_workspace_tensor(layout.py:256). #21996 then landed K3 code written against the old API (_k3_fp4_mla_cache.py:60,kernels/k3r_fp4_mla/fp4_mla_kv_append.py:46). All fourkimi_k3_nvfp4__sm_107__*targets import_fp4_mla_decode, so Kimi K3 NVFP4 on Rubin fails at import on main. The fix needs both parts: replace that import, and passshared=at_fp4_mla_decode.py:235,335,587,594,601and_k3_fp4_mla_cache.py:1519(the backend passesshared=Falsefor the packed plan atpacked_plan.py:342). Fixing only the import still fails this test with aTypeErrorfor the missingshared. Please route the bug to the #21996 owners with a priority. Also record the full error, because the truncated_fp#_mla_generati...signature reads like a name that does exist:ImportError: cannot import name '_fp4_mla_generation_num_blocks_device' from 'tensorrt_llm._torch.attention.backends.fp4_mla.metadata'.
| unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810) | ||
| unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506) | ||
| unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan SKIP (https://nvbugs/7113273) |
There was a problem hiding this comment.
[Nice to have] (R2: Test lists and waives) This line covers the file's only cpu_only case, but the file is also listed for GR100-PyTorch-Post-Merge-1 (l0_gr100.yml:127). There, test_step_writes_lengths_and_plan_once imports _fp4_mla_decode through _decode() and hits the same ImportError. On an SM107 runner with the CuTe DSL, so do the 16 test_decode_matches_backend_decode / test_reduce_pdl_is_bitwise cases. No post-merge run has included #21996 yet (L0_PostMerge 3079 built eafebd8), so the next one will fail them and file another bug for the same cause. Consider a file-level entry under this bug instead: unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py SKIP (https://nvbugs/7113273). apply_waives_ut falls back to the file key (tests/unittest/conftest.py:358-361).
| unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810) | ||
| unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506) | ||
| unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan SKIP (https://nvbugs/7113273) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) This is one of nine auto-waive PRs, under nine nvbugs, for one root cause. #21996's K3 code imports _fp4_mla_generation_num_blocks_device, which #22078 removed from the FP4 MLA backend; all nine share the same failure signature and an onset of 02:06-02:11Z. In L0_MergeRequest_PR 70869 that cause fails 28 cases on the CPU shards, and this PR waives 1 of them. The other 27 are waived only in #22523, #22525, #22534, #22536, #22539, #22542, #22543 and #22544, so #18873, #21835 and #22506 stay red after this PR alone. The six PRs that insert after line 170 also conflict with one another, and the eventual fix has to find lines under nine bug ids. Please fold them into one PR under one bug (mark the other eight as duplicates of 7113273), or link the nine PRs and merge them as a set.
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
Closing: nvbugs/7113273 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 1 failed case(s) on
all platformswith unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_planbroken on main for CPU: failure signatureimporterror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:07:03Z, latest failure 2026-11-18T04:39:25Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113273 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefixCPU), so every platform that runs the case fails it.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_planon CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
all platformsmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.09e4c2fb8-bdbc-405c-aea6-fce712ed5e78; snapshot SHA-256f7830b9927329ca0f0f6c0805f41a876bec82034b6dc60fbf0ec56dd9a83b619; config SHA-256e0a7823b8162d1429b790e7a890bf0c88023fb8b1475d03f4013949af585a936unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan: incident273607556aa61323938c9964a8e4080118f56c88068b23f894b6739975e994d7, observation71c17a6946dd54f11c986d21c73afacdfb8a2825c980480dd6c7bf8c0e4cee58Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt. No source code, configuration, or API changes.test_plan_geometry_follows_the_page_planon all platforms. The test does not run on any platform until the waiver is removed.QA Engineer Review
Test files and test-list files changed:
tests/integration/test_lists/waives.txt(+1 line, 0 removed).unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4mla_decode.py::test_plan_geometry_follows_the_page_planasSKIP (https://nvbugs/7113273).Behavior and coverage:
#18873,#21835, and#22506as affected. Those runs fail on this test.Coverage verdict:
needs follow-up. The waiver must be removed after bug 7113273 is resolved, and the test must pass on main.Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds a SKIP entry for one K3 FP4 MLA decode test. Verify that the test passes again and remove the entry after the linked bug is closed.