Skip to content

[https://nvbugs/7113273][infra] Waive 1 failed case(s) on all platforms - #22524

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/9e8eea9679b02e034c3a9221dbe48cd5
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/9e8eea9679b02e034c3a9221dbe48cd5

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on all platforms with unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan broken on main for CPU: failure signature importerror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:07:03Z, latest failure 2026-11-18T04:39:25Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113273 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefix CPU), so every platform that runs the case fails it.

  • Bug: https://nvbugs/7113273 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113273 is fixed or closed and the test passes post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T06:06:11.671555+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan SKIP (https://nvbugs/7113273)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope all platforms matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 9e4c2fb8-bdbc-405c-aea6-fce712ed5e78; snapshot SHA-256 f7830b9927329ca0f0f6c0805f41a876bec82034b6dc60fbf0ec56dd9a83b619; config SHA-256 e0a7823b8162d1429b790e7a890bf0c88023fb8b1475d03f4013949af585a936
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan: incident 273607556aa61323938c9964a8e4080118f56c88068b23f894b6739975e994d7, observation 71c17a6946dd54f11c986d21c73afacdfb8a2825c980480dd6c7bf8c0e4cee58

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • The change adds one skip entry to tests/integration/test_lists/waives.txt. No source code, configuration, or API changes.
  • The skip applies to test_plan_geometry_follows_the_page_plan on all platforms. The test does not run on any platform until the waiver is removed.
  • The objectives report a code-level import failure on main. The waiver does not fix that failure. It hides the failure from CI.
  • The objectives state that the proposal does not confirm that main is fixed. Restoration must be confirmed separately.
  • The entry references nvbug 7113273. The waiver must be removed after that bug is fixed or closed and the test passes.

QA Engineer Review

Test files and test-list files changed:

  • Modified: tests/integration/test_lists/waives.txt (+1 line, 0 removed).
  • Added waiver: unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4mla_decode.py::test_plan_geometry_follows_the_page_plan as SKIP (https://nvbugs/7113273).

Behavior and coverage:

  • The skipped test covers plan geometry for the K3 FP4 MLA decode path, which follows the page plan.
  • While the waiver is active, no CI run executes this test. Regressions in that path are not detected.
  • The objectives list pre-merge runs for #18873, #21835, and #22506 as affected. Those runs fail on this test.

Coverage verdict: needs follow-up. The waiver must be removed after bug 7113273 is resolved, and the test must pass on main.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds a SKIP entry for one K3 FP4 MLA decode test. Verify that the test passes again and remove the entry after the linked bug is closed.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 06:13
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration test skip list adds test_plan_geometry_follows_the_page_plan for K3 FP4 MLA decode. The entry references nvbug 7113273.

Changes

Skip-list update

Layer / File(s) Summary
Add test skip entry
tests/integration/test_lists/waives.txt
Adds test_plan_geometry_follows_the_page_plan to the skip list for K3 FP4 MLA decode. The entry references nvbug 7113273.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Merge Risk: Minimal · up to 0c3fe

This adds a temporary skip for one failing K3 FP4 MLA decode test, tracked in a bug. It has no runtime effect on production code, and removing the line is trivial once the bug is fixed.

Architecture risk: Low · up to 0c3fe

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Adds a skip entry for the K3 FP4 MLA decode page-plan geometry test, linked to nvbug 7113273.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the NVBug ID and the infra waiver type, which matches the change. The title is a single line that follows the repository template. It states the main change: waiving one failed test ca…
Description check Passed The description follows the repository template in outline. It gives a clear summary of the waiver, the affected test, the failing platform (CPU), the failure signature, the bug link, and the removal …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the NVBug ID and the infra waiver type, which matches the change. The title is a single line that follows the repository template. It states the main change: waiving one failed test case on all platforms. It does not name the test or the failure cause, which is acceptable for a title.

Full details: Description check

Explanation

The description follows the repository template in outline. It gives a clear summary of the waiver, the affected test, the failing platform (CPU), the failure signature, the bug link, and the removal condition. The Description section is filled in and the reasoning is clear. The Test Coverage section does not name a specific test that safeguards the change. It only asks reviewers to check the incident evidence. The PR Checklist is left unchecked, which the template allows. The description also includes a template-like auto-waive block, and the author left the template's CODING_GUIDELINES and related items unaddressed. Because the description is mostly complete, the check passes. The Test Coverage gap is minor for a waive-only change.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between ba78e87 and 0c3fe03.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers. Posted 3 comments, all Nice to have: 2 inline on tests/integration/test_lists/waives.txt:171 and 1 below. The waive itself is correct. The test fails on main and on the merge base with the ImportError described below (CPU-Generic-x86-3/arm-3 in L0_MergeRequest_PR 70869, 70902 and 70908), the entry matches how apply_waives_ut applies unittest waives, and it cites the bug filed for this failure.

  • [Nice to have] (N3: Missing or ineffective tests) For nvbugs/7113273 (no owner, Unprioritized): this waive removes the only CPU signal for a break that is not test-only. #22078 removed _fp4_mla_generation_num_blocks_device from fp4_mla/metadata.py and made shared a required keyword-only argument of _ensure_workspace_tensor (layout.py:256). #21996 then landed K3 code written against the old API (_k3_fp4_mla_cache.py:60, kernels/k3r_fp4_mla/fp4_mla_kv_append.py:46). All four kimi_k3_nvfp4__sm_107__* targets import _fp4_mla_decode, so Kimi K3 NVFP4 on Rubin fails at import on main. The fix needs both parts: replace that import, and pass shared= at _fp4_mla_decode.py:235,335,587,594,601 and _k3_fp4_mla_cache.py:1519 (the backend passes shared=False for the packed plan at packed_plan.py:342). Fixing only the import still fails this test with a TypeError for the missing shared. Please route the bug to the #21996 owners with a priority. Also record the full error, because the truncated _fp#_mla_generati... signature reads like a name that does exist: ImportError: cannot import name '_fp4_mla_generation_num_blocks_device' from 'tensorrt_llm._torch.attention.backends.fp4_mla.metadata'.

unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan SKIP (https://nvbugs/7113273)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R2: Test lists and waives) This line covers the file's only cpu_only case, but the file is also listed for GR100-PyTorch-Post-Merge-1 (l0_gr100.yml:127). There, test_step_writes_lengths_and_plan_once imports _fp4_mla_decode through _decode() and hits the same ImportError. On an SM107 runner with the CuTe DSL, so do the 16 test_decode_matches_backend_decode / test_reduce_pdl_is_bitwise cases. No post-merge run has included #21996 yet (L0_PostMerge 3079 built eafebd8), so the next one will fail them and file another bug for the same cause. Consider a file-level entry under this bug instead: unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py SKIP (https://nvbugs/7113273). apply_waives_ut falls back to the file key (tests/unittest/conftest.py:358-361).

unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_k3_fp4_mla_decode.py::test_plan_geometry_follows_the_page_plan SKIP (https://nvbugs/7113273)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) This is one of nine auto-waive PRs, under nine nvbugs, for one root cause. #21996's K3 code imports _fp4_mla_generation_num_blocks_device, which #22078 removed from the FP4 MLA backend; all nine share the same failure signature and an onset of 02:06-02:11Z. In L0_MergeRequest_PR 70869 that cause fails 28 cases on the CPU shards, and this PR waives 1 of them. The other 27 are waived only in #22523, #22525, #22534, #22536, #22539, #22542, #22543 and #22544, so #18873, #21835 and #22506 stay red after this PR alone. The six PRs that insert after line 170 also conflict with one another, and the eventual fix has to find lines under nine bug ids. Please fold them into one PR under one bug (mark the other eight as duplicates of 7113273), or link the nine PRs and merge them as a set.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113273 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants