Skip to content

[https://nvbugs/7113610][infra] Waive 1 failed case(s) on CPU - #22544

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/267e652b9db0619a19e5c42a61a67717
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/267e652b9db0619a19e5c42a61a67717

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] broken on main for CPU: failure signature assertionerror: tensorrt_llm._torch.attention.back...evice = <ast.alias object at #x#>.name, onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113610 tracks the fix. Scope CPU is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7113610 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113610 is fixed or closed and the test passes on CPU post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T07:06:07.420204+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope CPU matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 7ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-256 3b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]: incident 2481ff971662a7cec996889b7f819a4875af3a8d682e3251829daad8115ccc40, observation fd5ac07ee9742bcbf5c194b1a9eb03a953bf9a097cea2ee25bb0f26c47115868

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • The change adds one waiver line to tests/integration/test_lists/waives.txt. No source code, configuration, or test code changes.
  • The waiver skips test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] on the CPU stage. It links to NVBUG 7113610.
  • The waiver has no expiry or removal condition in the file. Removal depends on the bug being fixed or closed, and on a passing CPU run after merge. Someone must track this manually.
  • The objectives show the entry with an abbreviated path (CPU/...). Confirm the full selector matches the failing test ID exactly. A mismatch leaves the test running and failing.

QA Engineer Review

  • Changed test-list file: tests/integration/test_lists/waives.txt, +1 line.
  • Added waiver: full:CPU/...::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610).
  • Affected test: unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache].
  • Behavior covered: the test no longer runs on CPU. This test checks the import contract of the FP4 MLA cache copies from the backend. CPU coverage for this case is lost until the waiver is removed.
  • The change does not touch test-db/ or qa/ lists.
  • Coverage verdict: needs follow-up. Verify the selector matches the failing test ID. Remove the waiver after NVBUG 7113610 is fixed and the CPU test passes post-merge.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds a CPU skip for one KIMI K3 FP4 MLA copies import-contract test. QA should confirm the test passes on CPU once the bug is resolved and the waiver is removed.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 07:17
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration test waiver list adds a CPU skip entry for the Kimi K3 FP4 MLA cache backend-contract import test, referencing NVBUG 7113610.

Changes

CPU test waiver

Layer / File(s) Summary
Add CPU skip entry
tests/integration/test_lists/waives.txt
Adds a CPU skip entry for the specified backend-contract import test and references NVBUG 7113610.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Merge Risk: Minimal · up to 565e1

This is a waiver-only change that temporarily skips one CPU test case tracked by a bug. It has no production impact, and it is safe to merge once reviewers confirm the bug link.

Architecture risk: Low · up to 565e1

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added the CPU skip entry for the Kimi K3 FP4 MLA cache backend-contract import test, referencing NVBUG 7113610.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title follows the repository template with an NVBugs ID and the [infra] type. It names the main change, a CPU waiver for one failed test case. The title is clear and specific, and it matches the s…
Description check Passed The description states the purpose, the failing test, the NVBugs link, the scope (CPU), and the removal condition. It follows the template's Description and PR Checklist sections. The Test Coverage se…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title follows the repository template with an NVBugs ID and the [infra] type. It names the main change, a CPU waiver for one failed test case. The title is clear and specific, and it matches the single-line skip entry in waives.txt.

Full details: Description check

Explanation

The description states the purpose, the failing test, the NVBugs link, the scope (CPU), and the removal condition. It follows the template's Description and PR Checklist sections. The Test Coverage section is only a placeholder and does not name specific tests. The description also includes automated provenance details that are not part of the template. Overall, the description is mostly complete.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 0cea16a and 565e1ce.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 4 remain after this review.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@karljang

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85730 [ skip ] triggered by Bot. Commit: 565e1ce Link to invocation

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the one-line CPU waive. The entry is well formed and sorted, and its full:CPU/ prefix matches every CPU-Generic x86/arm stage that runs l0_cpu.yml. It skips only the failing parametrization (the other three still run and pass), and it cites the bug filed for this failure. Posted comments: 0 blockers, 0 possible blockers, 2 nice to have. Both are about merge coordination: the two sibling waive PRs for the same file, and the in-flight fix #22545.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R2: Test lists and waives) For nvbugs/7113610, the cause and the fix are already known. #21996 made _k3_fp4_mla_cache.py:60 and kernels/k3r_fp4_mla/fp4_mla_kv_append.py:46 import _fp4_mla_generation_num_blocks_device, which #22078 had removed from fp4_mla/metadata.py. The open #22545 defines that helper in _k3_fp4_mla_cache.py. With its diff applied to this base, all 8 cases of test_modeling_v2_k3_fp4_mla_copies.py pass, this one included.

#22545 does not touch waives.txt, though, so this line and the fix race. If #22545 merges first, please close this PR rather than merge a waive for a passing test. Otherwise, please have #22545 (or the bug owner) delete this line, and the sibling lines from #22543/#22536, in the same change. Please also link #22545 in the bug, which is still unprioritized.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache] SKIP (https://nvbugs/7113610)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) This is one of three open auto-waive PRs for the same file and the same root cause. #22543 (nvbugs/7113609) waives test_copied_definitions_match_their_sources[_k3_fp4_mla_cache], and #22536 (nvbugs/7113412) waives test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]. Running the file on this tree gives exactly these three failures (3 failed, 5 passed), all from #21996 landing after #22078 changed the FP4 MLA backend.

Merged alone, this PR does not turn the CPU stages green. The PRs listed in the description still fail on the other two cases, and so does this PR's own CPU run, because change-based selection runs the whole file. All three PRs also insert after line 69, so whichever merges second conflicts at this line: git merge-file of this PR with #22543 conflicts here.

Please land the three together, either folded into one PR or merged back to back with a rebase. Please also mark nvbugs/7113609, 7113412 and 7113610 as one root cause, so a single fix closes all three.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #85730 [ skip ] completed with state SUCCESS. Commit: 565e1ce
Skipping testing for commit 565e1ce

Link to invocation

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113610 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants