Skip to content

[https://nvbugs/7113412][infra] Waive 1 failed case(s) on CPU - #22536

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/0e558a73208bc9d284ac575aef3c96cf
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/0e558a73208bc9d284ac575aef3c96cf

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] broken on main for CPU: failure signature assertionerror: _k#_fp#_mla_kernels._fp#_mla_atten...on, <ast.functiondef object at #x#>)]}, onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113412 tracks the fix. Scope CPU is the stage prefix that ran the failing test.

  • Bug: https://nvbugs/7113412 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113412 is fixed or closed and the test passes on CPU post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T06:36:08.494883+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope CPU matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run b63a992e-b7c5-4815-92d3-fe7b0737fe16; snapshot SHA-256 5fb2f3fdddfe9ce39251cec27d66dc9e9c21975354c430fd53248a3ea4b859d8; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels]: incident 41da36aea844a2574e27179e79f8e7546e2a01f1fb0600a093ac95184ae2920a, observation 59881bcfdc1c01b507ada7ff7043613fac263c675bea789b372f41b7e6a477c0

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • The change adds one line to tests/integration/test_lists/waives.txt. It adds a CPU skip for unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels], linked to NVBUG 7113412.
  • No source code, API, or default changes are present in the diff.
  • Risk: the skip hides a failure that the incident evidence reports on main from 2026-11-18T02:11:36Z to 2026-11-18T04:58:01Z. The PR does not establish that main is fixed.
  • The waiver must be removed after the bug is fixed or closed and the test passes on CPU post-merge. Until then, the test does not run on CPU.

QA Engineer Review

  • Test-list file modified: tests/integration/test_lists/waives.txt (+1 line, 0 removed).
  • Waiver added: test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] on CPU, with reason https://nvbugs/7113412.
  • Behavior covered: the test checks that copied kernel definitions match their sources. With the waiver, CPU runs no longer check this case.
  • Test-function code, test-db (CI) lists, and qa lists are unchanged by this diff. Whether the test is listed in those lists is not established from this diff.
  • Coverage verdict: needs follow-up. QA should confirm the failure on main is resolved and remove the waiver when the bug is closed.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds a CPU skip for one kimi_k3_vl FP4 MLA copy-check case. QA should verify that the case passes on CPU once the bug is closed, then remove this entry.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 06:44
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The CPU skip list now includes test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels], tracked by NVBUG 7113412.

Changes

CPU skip-list update

Layer / File(s) Summary
Add CPU skip-list entry
tests/integration/test_lists/waives.txt
The CPU skip list adds the Kimi K3 FP4 MLA copied-definitions test, tracked by NVBUG 7113412.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Change: Other

Merge Risk: Minimal · up to 4668a

This adds a temporary CPU skip for one test that is failing on main, tracked by a bug. It does not change production behavior. The merge risk is minimal; the waiver should be removed when the bug is fixed.

Architecture risk: Low · up to 4668a

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added the Kimi K3 FP4 MLA copied-definitions test to the CPU skip list with NVBUG 7113412.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the NVBug ID and the [infra] type, and says a single failed case is waived on CPU. That matches the single-line waives.txt change that adds a CPU skip for the copied-definitions test. …
Description check Passed The description follows the repository's general layout. It explains the waiver, the failing test, the bug link, the scope (CPU), and the removal condition. It gives incident evidence and provenance. …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the NVBug ID and the [infra] type, and says a single failed case is waived on CPU. That matches the single-line waives.txt change that adds a CPU skip for the copied-definitions test. The title follows the repository template and is specific enough for history scanning.

Full details: Description check

Explanation

The description follows the repository's general layout. It explains the waiver, the failing test, the bug link, the scope (CPU), and the removal condition. It gives incident evidence and provenance. It leaves the Test Coverage section as a template note and does not list specific tests. The PR checklist is unchecked, and the template's 'Description' and 'Test Coverage' sections were partly filled in with auto-generated text rather than the author's own explanation.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between befcc89 and 4668a7a.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this one-line waive against the CI results of the runs listed in the description. The line is well formed and applies to every CPU-* stage, x86 and arm. It skips only a case that fails on main and cites the bug filed for it. Comments: 0 blockers, 0 possible blockers, 3 nice to have. They cover coordination with the sibling waive PRs #22543 and #22544, a merge conflict with current main, and the open fix #22545.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R2: Test lists and waives) This line waives 1 of the 3 cases of test_modeling_v2_k3_fp4_mla_copies.py that fail on main from the same root cause: #21996 copied the FP4 MLA backend as it was before #22078 removed _fp4_mla_attention_v_repack_kernel and _fp4_mla_generation_num_blocks_device. The other two cases fail in every run behind the PRs listed in the description (e.g. L0_MergeRequest_PR 70898, CPU-Generic-x86-1 and CPU-Generic-arm-1):

  • test_copied_definitions_match_their_sources[_k3_fp4_mla_cache]
  • test_copies_import_only_the_contract_from_the_backend[_k3_fp4_mla_cache]

They are waived separately, in #22543 (nvbugs/7113609) and #22544 (nvbugs/7113610). l0_cpu.yml lists this file as a whole, so this PR on its own leaves both CPU stages red, and the six PRs listed above still fail after a rebase. Suggest putting the three lines in one PR under one bug (marking 7113609/7113610 as duplicates of 7113412), or merging the three PRs together.

full:B300/accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_4gpus[v2_kv_cache-dp4-trtllm-auto] SKIP (https://nvbugs/7023202)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R2: Test lists and waives) For nvbugs/7113412, which has no engineer assigned yet: #22545, from the author of #21996, already fixes all three failing cases of this file and keeps their test ids. It does not reference this bug or 7113609/7113610, though, so nothing ties the fix to removing this line. Please assign the bug and link it from #22545:

Otherwise [_k3_fp4_mla_kernels] stays skipped after it passes again, and nothing on CPU checks the other 23 copied definitions in _k3_fp4_mla_kernels.py against the backend. FYI, the waived case has four orphaned definitions, not only the one in the failure message: the three _fp4_mla_attention_v_repack* kernels and fp4_mla_repack_v_cache.

full:B300/disaggregated/test_disaggregated.py::test_disaggregated_mamba_conc_greater_than_mbs[NVIDIA-Nemotron-3-Super-120B-A12B-FP8] SKIP (https://nvbugs/7088440)
full:B300/disaggregated/test_disaggregated.py::test_disaggregated_qwen3_32b_fp8[Qwen3/Qwen3-32B-FP8] SKIP (https://nvbugs/7099626)
full:CPU/unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_fp4_mla_copies.py::test_copied_definitions_match_their_sources[_k3_fp4_mla_kernels] SKIP (https://nvbugs/7113412)
full:CPU/unittest/_torch/modeling_v2/test_modeling_v2_ci_selection.py::test_every_test_is_in_a_test_list SKIP (https://nvbugs/7082290)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) This branch no longer merges into main (GitHub reports conflicts). #22510 (ceb406c) deleted this test_modeling_v2_ci_selection.py::test_every_test_is_in_a_test_list ... (https://nvbugs/7082290) line, right below the new waive. Please regenerate the branch on current main. #22543 and #22544 insert at the same position, so after any one of the three merges, the other two conflict again. One PR carrying all three lines avoids that.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113412 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants