Skip to content

[https://nvbugs/7113411][infra] Waive 1 failed case(s) on all platforms - #22539

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/5461c95322af8a56ad1d8d1b0fb2ea28
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/5461c95322af8a56ad1d8d1b0fb2ea28

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 1 failed case(s) on all platforms with unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107 broken on main for CPU: failure signature importerror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:07:18Z, latest failure 2026-11-18T05:13:25Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113411 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefix CPU), so every platform that runs the case fails it.

  • Bug: https://nvbugs/7113411 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113411 is fixed or closed and the test passes post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T06:36:08.494883+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107 on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107 SKIP (https://nvbugs/7113411)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope all platforms matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run b63a992e-b7c5-4815-92d3-fe7b0737fe16; snapshot SHA-256 5fb2f3fdddfe9ce39251cec27d66dc9e9c21975354c430fd53248a3ea4b859d8; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107: incident 9ac83197b44a5cd211800e64d1909e1669576b4df7fbf7345830d1c58524dfcc, observation fa25b86d133fa4ba146323147e318a33ccb1b89704debad7230af8d2dc99a8f0

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • No source code changes. The only changed file is tests/integration/test_lists/waives.txt (+1/-0).
  • The added entry is a waiver with no selector scope beyond the single test ID. It applies on all platforms and in every CI stage that runs this test.
  • The entry sets SKIP for test_route_returns_the_target_class_only_on_sm107. The failure is described as a code-level error. The skip hides that error until the bug is fixed.
  • The PR description says to remove the waiver after NVBUG 7113411 is fixed or closed and the test passes post-merge. The PR does not establish that main is fixed.

QA Engineer Review

  • Test-list file changed: tests/integration/test_lists/waives.txt.
  • Entry added: unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107 SKIP (https://nvbugs/7113411).
  • Test function removed or modified: none.
  • Behavior covered by the skipped test: SM107 routing for the Kimi K3 VL model, which returns the target class only on SM107. Coverage for this path is lost while the waiver is active.
  • Test-list placement: not applicable to this change. The entry is a waiver, not a new test selector.
  • Coverage verdict: needs follow-up. The waiver removes coverage. Follow-up should confirm the fix for NVBUG 7113411 and remove the entry once the test passes.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds one skip waiver for the SM107 Kimi K3 VL routing test. QA should confirm the test passes on SM107 after the fix and then remove this line.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 06:46
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The skip list now includes the SM107 Kimi K3 VL routing test, associated with NVBUG 7113411.

Changes

Routing test skip

Layer / File(s) Summary
Add routing test to skip list
tests/integration/test_lists/waives.txt
Adds the SM107 Kimi K3 VL routing test to the skip list under NVBUG 7113411.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Suggested reviewers: brnguyen2

Merge Risk: Minimal · up to 508d6

This adds one CI waiver for the SM107 Kimi K3 VL routing test, which currently fails from a missing import in production code. The waiver covers the failing path, so it should unblock affected CI runs. Coverage for that routing path is reduced until NVBUG 7113411 is fixed, so the waiver should be removed once the bug is resolved.

Architecture risk: Low · up to 508d6

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Adds the SM107 Kimi K3 VL routing test to the skip list, tracked by NVBUG 7113411.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the NVBugs ID and the infra type, which follows the repository template. It also states the main change, a waiver for one failed case on all platforms. A reviewer scanning history can …
Description check Passed The description follows most of the template. It explains the waiver, the failing test, the bug link, the removal condition, and the affected PRs. It also includes a provenance section and a pre-merge…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the NVBugs ID and the infra type, which follows the repository template. It also states the main change, a waiver for one failed case on all platforms. A reviewer scanning history can see the purpose. The title does not name the test file or the failure, but it does not need to.

Full details: Description check

Explanation

The description follows most of the template. It explains the waiver, the failing test, the bug link, the removal condition, and the affected PRs. It also includes a provenance section and a pre-merge checklist. The template's Description and Test Coverage sections are not filled in by the author. The Test Coverage section holds generic review notes, not a list of tests. The PR checklist box is unchecked, as the text asks for before merge.

Resolution

Optional: state the failure and the waiver scope in the Description section in the template's own words. Replace the generic Test Coverage text with the tests that cover the change, or state that none apply. Check the PR checklist after review.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between befcc89 and 508d664.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the one-line waive. It takes effect and matches the failure: the test fails on main in every CPU-Generic run since #21996 with the _fp4_mla_generation_num_blocks_device ImportError, it is cpu_only and listed only in l0_cpu.yml, the line is sorted and passes the test-list validator, and the bug link and restore condition are present. Posted comments: 0 blockers, 0 possible blockers, 1 nice to have.

unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/test_k3_sm107_routing.py::test_route_returns_the_target_class_only_on_sm107 SKIP (https://nvbugs/7113411)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) #22545 fixes this same failure. #22078 removed _fp4_mla_generation_num_blocks_device from fp4_mla/metadata.py, #21996's Kimi K3 code still imports it (_k3_fp4_mla_cache.py:58, k3r_fp4_mla/fp4_mla_kv_append.py:46), and #22545 moves the helper into _k3_fp4_mla_cache.py. Eight sibling auto-waive PRs (#22523, #22524, #22525, #22534, #22536, #22542, #22543, #22544) waive other tests that fail with the same ImportError, each under its own NVBug. None of these PRs or bugs is linked to the fix. If #22545 lands first and this merges afterwards, the line skips a passing test, which is the only CI check that K3 routing.route returns the SM107 target class. The restore condition keys only on nvbugs/7113411, so nothing prompts removing the line.

Suggest linking nvbugs/7113411 and the sibling bugs to #22545 and settling the merge order. If the fix lands first, close this PR. Otherwise, #22545 or the bug owner should delete this line together with the siblings' lines. Note that the PRs listed in the description need #22545 regardless, because the same ImportError aborts collection of three SM107 test directories, and no waive line can skip a collection error.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113411 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants