Repository navigation
[https://nvbugs/7113764][infra] Waive 20 failed case(s) on all platforms - #22548
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe integration skip list adds 20 waivers for Kimi K3 VL MLA attention operator tests. All entries reference NVBUG 7113764. ChangesKimi K3 VL MLA test waivers
Estimated code review effort: 1 (Trivial) | ~3 minutes Change: Other Merge Risk:
|
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
/bot skip --comment "auto-waive: waive-only change" |
|
PR_Github #85738 [ skip ] triggered by Bot. Commit: |
trtllm-agent-review
left a comment
There was a problem hiding this comment.
Thanks for the quick waive. Posted: 1 blocker (inline) and 1 nice to have (below). The 20 entries are well-formed: they match the collected node ids, sit in sorted order with no duplicates, cite nvbugs/7113764, and the unscoped form fits a code-level ImportError in a file that only l0_b300.yml runs. But one more parametrization of the same test fails with the same error and is not waived, so DGX_B300-PyTorch-3 stays red after this merges.
- [Nice to have] (R1: PR process) For nvbugs/7113764: the root-cause fix is already open as #22545 (it restores
_fp4_mla_generation_num_blocks_devicein_k3_fp4_mla_cache.py), but it neither cites the bug nor removes these lines, and this PR does not mention it. CI merges the TOT waive list into every PR run and only drops the lines a PR itself deletes (jenkins/L0_MergeRequest.groovy:811-835). So please link #22545 to the bug and agree on the order: if #22545 merges first, close this PR; otherwise #22545 should delete these lines in its own diff, so that its DGX_B300 run actually exercises the fix.
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32] SKIP (https://nvbugs/7113764) |
There was a problem hiding this comment.
[Blocker] (B3: CI breakage) test_context_and_decode_match_backend[fp8-op_b2_q1-128] is missing here, and it fails on DGX_B300 with the same ImportError. The test calls ops = _ops() at test_modeling_v2_k3_mla_attn_ops.py:415, before its off-SM107 pytest.skip at lines 419-425. _ops() goes through k3_mla_cache.py:40 to _k3_fp4_mla_cache.py:58-64 and raises ImportError: cannot import name '_fp4_mla_generation_num_blocks_device', so on SM103 this case errors out instead of skipping.
In L0 #70898 (PR #22261), DGX_B300-PyTorch-3 reports this file as tests=22, failures=21, skipped=1, and [fp8-op_b2_q1-128] is one of the 21. L0 #70869, #70899, #70903, #70904, #70907 and #70910 show the same result. With 20 of the 21 waived, DGX_B300-PyTorch-3 stays red for every PR (CI merges the TOT waive list into each run), so the 7 PRs listed in the description still fail when they re-run.
Suggested fix: add the case at its sorted position, right above this line:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-128] SKIP (https://nvbugs/7113764)
and say 21 cases in the title and description. Alternatively, replace the 20 lines with one file-level entry, unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py SKIP (https://nvbugs/7113764). That works for three reasons:
- on
l0_b300.yml:221, the only list that runs this file, every case either fails with this ImportError or skips itself (test_nvfp4_generation_scatter_matches_backendis SM107-only); - the unittest waive matcher supports file-level entries (
tests/unittest/conftest.py:346-361, precedent atwaives.txt:207-208); - the un-waive becomes a single line.
|
PR_Github #85738 [ skip ] completed with state |
|
Closing: nvbugs/7113764 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Pull request was closed
Description
Temporarily waive 20 failed case(s) on
all platformswith unscoped line(s): the failure is a code-level error that fails on every platform, first seen on DGX_B300. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128]broken on main for DGX_B300: failure signatureimporterror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:58:27Z, latest failure 2026-11-18T05:18:29Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113764 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on DGX_B300; stage prefixDGX_B300), so every platform that runs the case fails it.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backendon DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backendon DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8]on DGX_B300unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4]on DGX_B300PRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
all platformsmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.068f6b894-5303-4cc5-b987-62d006892bb6; snapshot SHA-2561bd0b0533cd1b46b9c51ed772e92e6f8de5fdbb6c5359a95f9b66ab9c16dd720; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128]: incident38f52f8de0a96813193d7788f6896366837ecff6b54bff3be22eb5b571460dd4, observationbdcf369d10762ef450ea192c55e055928a6a23d1d6531d554314e1346a06279aunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32]: incident8f470fb62e9e9cb801a7516e9479255a161222bf1bab894d229c1cfbef1674dd, observation18d51459b13dfdcb9135a93f3e31364ce5791646f2b2643ac99cae0c97071432unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128]: incident1cbca69c1e6050c6e9a014b6b3cdaa76ba304ea293f7537994d5b740bc932562, observation9fbf82e34e651ae1b39bafcf580563291e5f9b1cd8f79e50a6f370c3a9847ac4unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32]: incidenta63e6288f82fa47ecac2caba203c1a253cca5b2b8c3ce370d096991217d6f864, observation6fe3dbf1ee259a2efca3e0e1be0de4ee27d8b493f13e34e7e635a3e57570a140unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128]: incident3a7473a3ea8f4e26fb86763bea0e07849279d645f4b596915ee6a79e3cf90d45, observation13472bc354b60cc775faec2aa5b5335c81e15ab95a907165c636f844a7b3f419unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32]: incident5249d9e54bc94b012c58697ed7f40e2b86f23015916a707f0c78b0b61d4c3cfc, observation6c74069a717966d8a595ab9b73bf6675e78786992bd4d8a70395c11d3e7d8624unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128]: incidentbecfcdc10ffb188b30a88235f0d2a2e1bf92ebcf641c097e17b0234300a904a2, observation027ef0f79c3e5eb21b89f5bb0708a3b98fa28986bae5fbe4019c0d0f3e8589dbunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32]: incident81e4bd69c324a6a8bf12d1aa9fc7e004d03bc4a2f0df1eaf56422855f1397573, observation03a4ce16597ecf78fb3e0ba70333c985e129662b7008444dddfe193b43e428b9unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128]: incident7d2f4cae2251f772a138b5a3a918d1ef2f09edba3563a0c72eece345bfd00b3c, observationb12d36f30c52f849654eeae8f33d67bd817a7a747ff65823abd937daf833222eunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32]: incidente1e9e93baa20dac8732bfed345fa69dba2a3a00a09c1ea2a66c45fea77e4685c, observationae88329e730dda8d56619f3d9e992c706c1810cde944b229054e1e889585fb99unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32]: incident19addd74af0a6a17d4b594ff88692f290f797a52602a51109a5511cbd7519374, observation9bc65d26bd753341df79090f101130202a9132e1df7caac36f58b2f11300f133unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv]: incident56c9593fc5d486381fb5ce9983e0c56fcbf5cef2660ec2ff1d249dfa32ae3a9e, observationb5f60336a4da5affc0b0ce1b00e24586f38e3c41dd29857c2853ae73a2381c5cunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill]: incidentc895b8a4e25d54dd3866eb88f50320ce5456b05adc4851e05c5b9fa7ece7a2a7, observationd296f88a384f46b29a2d85fabf2842fa049b31f27cc5285cff1576ae5a040cf2unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv]: incident7bae9f70c9c3e27734dbb53271e51e67a72550a872e9478a8aed1c86963c4fee, observationf4cf96e6c5c38c6b25335118a797d0cd2d83c224d31da60a11c731c91317c272unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill]: incidentbd354ccba6069173641e20f2f2fa6aaf81c994c737c3cb901ce4d67817d21f8c, observationd3e56f178f5e89db6e01f7d714972935a6871a2951232bf06d6e1453da7caa7eunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backend: incident72f73eb6f8b8bc1634633b042570d5e99e8abb87c6c282e71d8b2d980d76d8b3, observation12a230749ff0dcc0d75393d90583d10a9b1344c96f9d61230c5c65f81adba7f6unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backend: incident0fb54ddd7623d2a97b7f6f1302e77b0d97af89d5a510738998b33393ff1a2556, observationdd4029b2e244a191d4ffeb9d43a726c9d0d0c97bcbe7cc819820a2f14f86022cunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16]: incident9cd24e53bb47eb4099f9b94f649d0d1c5c43e0e18b25ba0436a9437869a80638, observationad109cd06d87d8269525bdc481df03e0bd53b281b2378776af7dc0b11b78cfbaunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8]: incident626406ad0e3da6c629abde891db86faea807c320e20b8f914563531842fa4c35, observation9523358be72d219a34e838b7b3a23c7849e7a533ef2f8469dd3aab2c4d1868beunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4]: incidente4f7bda96ea05d99eafcddb77d08945aec0e83e65bdf5eb5d4ae021e43a94628, observation9b310649af7652a0f24db48f969f25c09c8eb377063fca4db908a35f6d06c3b7Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt(+20 lines, 0 removed).SKIPwaiver entries for Kimi K3 VL MLA attention tests underunittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py. Each entry references https://nvbugs/7113764.QA Engineer Review
Test-list files changed:
tests/integration/test_lists/waives.txt(+20 entries, allSKIP, all bound to NVBUG 7113764).Waived test IDs, grouped by test function:
test_context_and_decode_match_backend: 12 parametrized cases acrossbf16andfp8withcute_table_b8_q1,cute_wide_verify_b2_q2, andop_b2_q1variants, at sizes 32 and 128. Thefp8-op_b2_q1-128case is not in the list.test_fp8_cached_prefix_context_matches_backend: 4 cases (bf16_qkv/fp8_qkvwithcached_kv/chunked_prefill).test_nvfp4_context_matches_backend: 1 case.test_nvfp4_split_k_context_append_and_prefix_gather_match_backend: 1 case.test_state_matches_backend_roles: 3 cases (bf16,fp8,nvfp4).Coverage impact:
test-db/orqa/lists is not available from this diff.Coverage verdict: needs follow-up. The waiver is tracked, but the underlying failure is unresolved and no fix or passing run is reported.
Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds 20 skip entries for Kimi K3 MLA attention tests on all platforms. QA should confirm these tests pass once bug 7113764 is fixed, then remove the entries.