Skip to content

[https://nvbugs/7113764][infra] Waive 20 failed case(s) on all platforms - #22548

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/7afcd39eecdacbfddd7f7625b361722a
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/7afcd39eecdacbfddd7f7625b361722a

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 20 failed case(s) on all platforms with unscoped line(s): the failure is a code-level error that fails on every platform, first seen on DGX_B300. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128] broken on main for DGX_B300: failure signature importerror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:58:27Z, latest failure 2026-11-18T05:18:29Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113764 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on DGX_B300; stage prefix DGX_B300), so every platform that runs the case fails it.

  • Bug: https://nvbugs/7113764 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113764 is fixed or closed and the test passes post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T07:36:07.783679+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backend on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backend on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8] on DGX_B300
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4] on DGX_B300

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backend SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backend SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4] SKIP (https://nvbugs/7113764)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope all platforms matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 68f6b894-5303-4cc5-b987-62d006892bb6; snapshot SHA-256 1bd0b0533cd1b46b9c51ed772e92e6f8de5fdbb6c5359a95f9b66ab9c16dd720; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128]: incident 38f52f8de0a96813193d7788f6896366837ecff6b54bff3be22eb5b571460dd4, observation bdcf369d10762ef450ea192c55e055928a6a23d1d6531d554314e1346a06279a
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32]: incident 8f470fb62e9e9cb801a7516e9479255a161222bf1bab894d229c1cfbef1674dd, observation 18d51459b13dfdcb9135a93f3e31364ce5791646f2b2643ac99cae0c97071432
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128]: incident 1cbca69c1e6050c6e9a014b6b3cdaa76ba304ea293f7537994d5b740bc932562, observation 9fbf82e34e651ae1b39bafcf580563291e5f9b1cd8f79e50a6f370c3a9847ac4
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32]: incident a63e6288f82fa47ecac2caba203c1a253cca5b2b8c3ce370d096991217d6f864, observation 6fe3dbf1ee259a2efca3e0e1be0de4ee27d8b493f13e34e7e635a3e57570a140
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128]: incident 3a7473a3ea8f4e26fb86763bea0e07849279d645f4b596915ee6a79e3cf90d45, observation 13472bc354b60cc775faec2aa5b5335c81e15ab95a907165c636f844a7b3f419
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32]: incident 5249d9e54bc94b012c58697ed7f40e2b86f23015916a707f0c78b0b61d4c3cfc, observation 6c74069a717966d8a595ab9b73bf6675e78786992bd4d8a70395c11d3e7d8624
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128]: incident becfcdc10ffb188b30a88235f0d2a2e1bf92ebcf641c097e17b0234300a904a2, observation 027ef0f79c3e5eb21b89f5bb0708a3b98fa28986bae5fbe4019c0d0f3e8589db
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32]: incident 81e4bd69c324a6a8bf12d1aa9fc7e004d03bc4a2f0df1eaf56422855f1397573, observation 03a4ce16597ecf78fb3e0ba70333c985e129662b7008444dddfe193b43e428b9
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128]: incident 7d2f4cae2251f772a138b5a3a918d1ef2f09edba3563a0c72eece345bfd00b3c, observation b12d36f30c52f849654eeae8f33d67bd817a7a747ff65823abd937daf833222e
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32]: incident e1e9e93baa20dac8732bfed345fa69dba2a3a00a09c1ea2a66c45fea77e4685c, observation ae88329e730dda8d56619f3d9e992c706c1810cde944b229054e1e889585fb99
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32]: incident 19addd74af0a6a17d4b594ff88692f290f797a52602a51109a5511cbd7519374, observation 9bc65d26bd753341df79090f101130202a9132e1df7caac36f58b2f11300f133
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv]: incident 56c9593fc5d486381fb5ce9983e0c56fcbf5cef2660ec2ff1d249dfa32ae3a9e, observation b5f60336a4da5affc0b0ce1b00e24586f38e3c41dd29857c2853ae73a2381c5c
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill]: incident c895b8a4e25d54dd3866eb88f50320ce5456b05adc4851e05c5b9fa7ece7a2a7, observation d296f88a384f46b29a2d85fabf2842fa049b31f27cc5285cff1576ae5a040cf2
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv]: incident 7bae9f70c9c3e27734dbb53271e51e67a72550a872e9478a8aed1c86963c4fee, observation f4cf96e6c5c38c6b25335118a797d0cd2d83c224d31da60a11c731c91317c272
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill]: incident bd354ccba6069173641e20f2f2fa6aaf81c994c737c3cb901ce4d67817d21f8c, observation d3e56f178f5e89db6e01f7d714972935a6871a2951232bf06d6e1453da7caa7e
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backend: incident 72f73eb6f8b8bc1634633b042570d5e99e8abb87c6c282e71d8b2d980d76d8b3, observation 12a230749ff0dcc0d75393d90583d10a9b1344c96f9d61230c5c65f81adba7f6
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backend: incident 0fb54ddd7623d2a97b7f6f1302e77b0d97af89d5a510738998b33393ff1a2556, observation dd4029b2e244a191d4ffeb9d43a726c9d0d0c97bcbe7cc819820a2f14f86022c
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16]: incident 9cd24e53bb47eb4099f9b94f649d0d1c5c43e0e18b25ba0436a9437869a80638, observation ad109cd06d87d8269525bdc481df03e0bd53b281b2378776af7dc0b11b78cfba
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8]: incident 626406ad0e3da6c629abde891db86faea807c320e20b8f914563531842fa4c35, observation 9523358be72d219a34e838b7b3a23c7849e7a533ef2f8469dd3aab2c4d1868be
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4]: incident e4f7bda96ea05d99eafcddb77d08945aec0e83e65bdf5eb5d4ae021e43a94628, observation 9b310649af7652a0f24db48f969f25c09c8eb377063fca4db908a35f6d06c3b7

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • No source code changes. The only changed file is tests/integration/test_lists/waives.txt (+20 lines, 0 removed).
  • The change adds 20 SKIP waiver entries for Kimi K3 VL MLA attention tests under unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py. Each entry references https://nvbugs/7113764.
  • Scope: the entries use test IDs without platform qualifiers, so the waiver applies on all platforms. The PR objectives state this is intentional.
  • Risk: the waivers hide 20 failing tests without a fix. The PR objectives state the failure is an import error that is code-level, and that the waiver must be removed once bug 7113764 is fixed or closed. No fix is included in this PR.
  • The PR objectives say related pre-merge runs may need rebasing or rerunning after the waiver merges. No validation result is reported.

QA Engineer Review

Test-list files changed:

  • Modified: tests/integration/test_lists/waives.txt (+20 entries, all SKIP, all bound to NVBUG 7113764).

Waived test IDs, grouped by test function:

  • test_context_and_decode_match_backend: 12 parametrized cases across bf16 and fp8 with cute_table_b8_q1, cute_wide_verify_b2_q2, and op_b2_q1 variants, at sizes 32 and 128. The fp8-op_b2_q1-128 case is not in the list.
  • test_fp8_cached_prefix_context_matches_backend: 4 cases (bf16_qkv/fp8_qkv with cached_kv/chunked_prefill).
  • test_nvfp4_context_matches_backend: 1 case.
  • test_nvfp4_split_k_context_append_and_prefix_gather_match_backend: 1 case.
  • test_state_matches_backend_roles: 3 cases (bf16, fp8, nvfp4).

Coverage impact:

  • These tests cover the Kimi K3 MLA context and decode backend comparisons, cached-prefix context, NVFP4 context and split-K paths, and state-role handling.
  • The waiver removes CI signal for these paths until bug 7113764 is resolved.
  • Whether these tests appear in test-db/ or qa/ lists is not available from this diff.

Coverage verdict: needs follow-up. The waiver is tracked, but the underlying failure is unresolved and no fix or passing run is reported.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds 20 skip entries for Kimi K3 MLA attention tests on all platforms. QA should confirm these tests pass once bug 7113764 is fixed, then remove the entries.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 07:41
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration skip list adds 20 waivers for Kimi K3 VL MLA attention operator tests. All entries reference NVBUG 7113764.

Changes

Kimi K3 VL MLA test waivers

Layer / File(s) Summary
Add attention operator test waivers
tests/integration/test_lists/waives.txt
Adds skip entries for backend comparisons, cached-prefix context cases, NVFP4 context cases, and state-role cases. All entries reference NVBUG 7113764.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Merge Risk: Minimal · up to 7f46a

This PR temporarily skips 20 Kimi K3 MLA attention tests that fail with an import error on main, and it links them to a tracked bug. It has no production impact. Coverage returns once the bug is fixed and the waiver is removed, so the merge risk is minimal.

Architecture risk: Low · up to 7f46a

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added 20 skip-list entries for Kimi K3 VL MLA attention operator tests, all linked to NVBUG 7113764. They cover BF16/FP8 backend comparisons with varied configurations, cached-prefix context cases, NVFP4 context cases, and BF16/FP8/NVFP4 state-role cases.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the NVBugs ID and type tag, and says the change waives 20 failed cases on all platforms. That matches the only file change, which adds 20 skip entries to waives.txt. The title is speci…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the repository template in substance. It states the waiver's purpose, the failing tests, the bug link (nvbugs 7113764), the scope (unscoped, all platforms), and the removal con…
Full details: Title check

Explanation

The title names the NVBugs ID and type tag, and says the change waives 20 failed cases on all platforms. That matches the only file change, which adds 20 skip entries to waives.txt. The title is specific and on topic, so it passes.

Full details: Description check

Explanation

The description follows the repository template in substance. It states the waiver's purpose, the failing tests, the bug link (nvbugs 7113764), the scope (unscoped, all platforms), and the removal condition. The Description section is brief. The Test Coverage section does not list specific tests that safeguard the change. It only points to incident evidence and the selector scope. The PR Checklist has only the auto-filled checkbox ticked. The title is not shown in the description, so the template title format cannot be verified here. Most required information is present, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 8f09f7f and 7f46a78.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@lori-ren

Copy link
Copy Markdown
Contributor

/bot skip --comment "auto-waive: waive-only change"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85738 [ skip ] triggered by Bot. Commit: 7f46a78 Link to invocation

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick waive. Posted: 1 blocker (inline) and 1 nice to have (below). The 20 entries are well-formed: they match the collected node ids, sit in sorted order with no duplicates, cite nvbugs/7113764, and the unscoped form fits a code-level ImportError in a file that only l0_b300.yml runs. But one more parametrization of the same test fails with the same error and is not waived, so DGX_B300-PyTorch-3 stays red after this merges.

  • [Nice to have] (R1: PR process) For nvbugs/7113764: the root-cause fix is already open as #22545 (it restores _fp4_mla_generation_num_blocks_device in _k3_fp4_mla_cache.py), but it neither cites the bug nor removes these lines, and this PR does not mention it. CI merges the TOT waive list into every PR run and only drops the lines a PR itself deletes (jenkins/L0_MergeRequest.groovy:811-835). So please link #22545 to the bug and agree on the order: if #22545 merges first, close this PR; otherwise #22545 should delete these lines in its own diff, so that its DGX_B300 run actually exercises the fix.

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32] SKIP (https://nvbugs/7113764)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] (B3: CI breakage) test_context_and_decode_match_backend[fp8-op_b2_q1-128] is missing here, and it fails on DGX_B300 with the same ImportError. The test calls ops = _ops() at test_modeling_v2_k3_mla_attn_ops.py:415, before its off-SM107 pytest.skip at lines 419-425. _ops() goes through k3_mla_cache.py:40 to _k3_fp4_mla_cache.py:58-64 and raises ImportError: cannot import name '_fp4_mla_generation_num_blocks_device', so on SM103 this case errors out instead of skipping.

In L0 #70898 (PR #22261), DGX_B300-PyTorch-3 reports this file as tests=22, failures=21, skipped=1, and [fp8-op_b2_q1-128] is one of the 21. L0 #70869, #70899, #70903, #70904, #70907 and #70910 show the same result. With 20 of the 21 waived, DGX_B300-PyTorch-3 stays red for every PR (CI merges the TOT waive list into each run), so the 7 PRs listed in the description still fail when they re-run.

Suggested fix: add the case at its sorted position, right above this line:

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-128] SKIP (https://nvbugs/7113764)

and say 21 cases in the title and description. Alternatively, replace the 20 lines with one file-level entry, unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py SKIP (https://nvbugs/7113764). That works for three reasons:

  • on l0_b300.yml:221, the only list that runs this file, every case either fails with this ImportError or skips itself (test_nvfp4_generation_scatter_matches_backend is SM107-only);
  • the unittest waive matcher supports file-level entries (tests/unittest/conftest.py:346-361, precedent at waives.txt:207-208);
  • the un-waive becomes a single line.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85738 [ skip ] completed with state SUCCESS. Commit: 7f46a78
Skipping testing for commit 7f46a78

Link to invocation

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113764 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

auto-merge was automatically disabled November 18, 2026 17:07

Pull request was closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants