Skip to content

[https://nvbugs/7113609][infra] Waive 1 failed case(s) on CPU - #22543

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/11e9a4219b7b623fb3dcb7cd7f87da4e
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/11e9a4219b7b623fb3dcb7cd7f87da4e

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] broken on main for CPU: failure signature assertionerror: _k#_fp#_mla_cache.can_fuse_fp#_mla...fers from its source in [cache_update] assert [], onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113609 tracks the fix. Scope CPU is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7113609 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113609 is fixed or closed and the test passes on CPU post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T07:06:07.420204+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope CPU matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 7ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-256 3b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]: incident 4234c53ee1a8054e7858cc0d6e6f9d4fb24cb027dbf7121bdb18a3a9b8cf604c, observation 309e44f19e4e1a26054a4c3d1977d265cf7751f89ec0821c0e54fc2a1ef36094

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • tests/integration/test_lists/waives.txt gains one line at line 70. The line adds a CPU skip entry for test_copied_definitions_match_their_sources[_k3_fp4_mla_cache], tracked by nvbugs/7113609.
  • The change is a waiver only. It modifies no source code, configuration, or test logic.
  • The waiver hides a failure reported on main in copied definitions that differ from their source in cache_update. The PR does not establish that main is fixed. The waiver stays in place until the bug is fixed or closed and the test passes on CPU post-merge.
  • The PR description asks for removal of the waiver after that condition holds. No removal plan is attached in the diff.

QA Engineer Review

Test-list files changed:

  • Modified: tests/integration/test_lists/waives.txt (+1/-0).
  • Added waiver: full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609).
  • No test functions, source files, or test-db/ or qa/ lists changed.
  • The skipped test does not run on CPU while the waiver is active. The main failure in cache_update stays unverified by CI until the waiver is removed.
  • Coverage verdict: needs follow-up. Remove the waiver after the bug is fixed and the test passes on CPU.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds a CPU skip for test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]. QA should confirm that the test passes on CPU once nvbugs/7113609 is resolved, then remove this entry.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 07:14
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The CPU test waiver list now skips test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] and references NV bug 7113609.

Changes

CPU Test Waiver

Layer / File(s) Summary
Add CPU skip entry
tests/integration/test_lists/waives.txt
Adds a CPU skip-list entry for the copied-definition test and references NV bug 7113609.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Merge Risk: Minimal · up to 180fc

This temporarily skips one failing CPU copied-definition test and links it to a tracking bug. Production behavior is unchanged. It is low risk to merge, but the waiver should be removed once the bug is fixed.

Architecture risk: Low · up to 180fc

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added a skip-list entry for the CPU _k3_fp4_mla_cache copied-definition test, referencing NV bug 7113609.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title uses the required NVBugs ID and [infra] type format and states the main change: a temporary waiver of one failing CPU test case. It matches the single-line waives.txt change, so the check pa…
Description check Passed The description follows the template structure in part. It explains the waiver, the bug link, the scope, and the restore condition. It omits the template's Test Coverage content, which only has a plac…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title uses the required NVBugs ID and [infra] type format and states the main change: a temporary waiver of one failing CPU test case. It matches the single-line waives.txt change, so the check passes.

Full details: Description check

Explanation

The description follows the template structure in part. It explains the waiver, the bug link, the scope, and the restore condition. It omits the template's Test Coverage content, which only has a placeholder reviewer note, and the unchecked PR checklist box is left for the reviewer. The core rationale is clear, so the check passes with minor gaps.

Resolution

Optionally replace the placeholder Test Coverage text with the relevant test that covers the waiver, such as the test_copied_definitions_match_their_sources case, and state the expected CPU result once the bug is fixed.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between f1856a4 and 180fc6e.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 4 remain after this review.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The waive itself is correct. The id resolves, and the full:CPU/ prefix matches the CPU-Generic x86 and arm stages, which are the only stages that run this file (l0_cpu.yml:146). The case fails deterministically on f1856a4134 and on current main b5692a546f, with the signature in the description, and the line cites the bug filed for it. Posted 2 comments: 0 Blocker, 0 Possible blocker, 2 Nice to have. Both are about coordinating with the sibling waive PRs for the same file and with the open fix #22545.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) This is one of three cases of test_modeling_v2_k3_fp4_mla_copies.py that fail on CPU for the same reason: #21996 copied the FP4 MLA backend definitions from before #22078 changed them. The other two are waived in separate open PRs that insert at this same line: #22536 (test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels], nvbugs/7113412) and #22544 (test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache], nvbugs/7113610). Whichever lands second will conflict here. Until all three are in, the file's l0_cpu entry stays red on CPU-Generic x86/arm, because all three cases fail on f1856a4134 and on main b5692a546f. Suggest landing the three lines together (one PR, or back to back with a rebase) and marking the three bugs as duplicates, so one fix closes them all.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113609)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) For nvbugs/7113609: the open fix #22545 makes this case pass; with its changes to the K3 copies and to this test, all 8 cases of the file pass. It does not touch waives.txt or reference this bug, though. If this waive lands first, please have #22545 or a follow-up delete this line, plus the sibling lines from #22536/#22544, and link the bugs. Otherwise the parity check of all 33 copied definitions in _k3_fp4_mla_cache.py stays skipped after the fix. If #22545 lands first, this PR can just be closed. Also, the case stops at its first mismatch, so the bug's scope is wider than its title: 13 of the 33 definitions no longer match a backend source, not only can_fuse_fp4_mla_q_quant.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113609 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants