Skip to content

[https://nvbugs/7113611][infra] Waive 4 failed case(s) on all platforms - #22542

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/294dc95c47e22a1ce2cc1bba99ec356f
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/294dc95c47e22a1ce2cc1bba99ec356f

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 4 failed case(s) on all platforms with unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling] broken on main for CPU: failure signature importerror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113611 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefix CPU), so every platform that runs the case fails it.

  • Bug: https://nvbugs/7113611 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113611 is fixed or closed and the test passes post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T07:06:07.420204+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling] on CPU
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moeep8.modeling] on CPU
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moetp2ep4.modeling] on CPU
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__tp1.modeling] on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling] SKIP (https://nvbugs/7113611)
unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moeep8.modeling] SKIP (https://nvbugs/7113611)
unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moetp2ep4.modeling] SKIP (https://nvbugs/7113611)
unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__tp1.modeling] SKIP (https://nvbugs/7113611)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope all platforms matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run 7ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-256 3b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling]: incident e86058814659f0b540cce18fbe7b2c65647cf27d17a9d702a16fca0d500f1e9d, observation 97143d1897944af407f39549aa82ca6e88f59f36b5afb31adf6793f4758b592f
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moeep8.modeling]: incident ea18b5709e89ca6bdbf5a769103ef8e167c571a4f47aff87ef2aa6eeb3b91623, observation f78e202d9d825600318aa21ba8f41e328c50eca83d63bdbf16907abf4ad41db1
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moetp2ep4.modeling]: incident cd50fa2c313670400c01e00087f5e3cde567eef43f92fbc9cdd51c887f142929, observation 288e025ce777474ccf80bdedbb94e5bf87a0825fad9c6d5df7ebe90ec7f84cd6
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__tp1.modeling]: incident 59beeef11201710177b6a1e01af707292129846ee479583032f334dc0ee8854d, observation cddbbba3af55323fcd55a6a5a6340589a28c3301f6626297691adb8b6fb2b332

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • tests/integration/test_lists/waives.txt: adds four SKIP waiver entries. Each entry targets a test_sm107_glue_uses_the_family_metadata case in unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py. The four parameter IDs are kimi_k3_nvfp4__sm_107__dep16, helix8_moeep8, helix8_moetp2ep4, and tp1. All four reference https://nvbugs/7113611.
  • No source or test code changes. The waivers apply to all platforms, not only CPU.
  • The PR description says the cases fail on CPU with an import error. This diff does not show the error. The import failure is not verified from this change.
  • Scope: the waivers hide the cases on every platform. The description says to remove them after the bug is fixed or closed and the tests pass on main. This PR does not establish that main is fixed.

QA Engineer Review

  • Test-list file modified: tests/integration/test_lists/waives.txt (+4 lines, 0 removed).
  • Entries added: four SKIP waivers for test_sm107_glue_uses_the_family_metadata with the four parameter IDs listed above, all tied to NVBUG 7113611.
  • Covered behavior: none. The waivers disable these cases. No test code changes.
  • Regressions: the four cases no longer run in any CI stage that uses this list. Failures in the SM107 Kimi K3 VL KDA metadata path would not be reported while the waivers remain.
  • Test-db or QA list placement: not applicable to a waiver-only change.
  • Coverage verdict: needs follow-up. Confirm the bug status and the CPU import error. Remove the waivers once the bug closes and the tests pass.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: adds four SKIP waivers for the SM107 KDA metadata cases on all platforms. A QA engineer should confirm the cases pass after the bug is fixed and then remove these entries.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 07:11
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The skip list now includes four SM107 Kimi K3 VL KDA metadata test configurations. Each entry references NVBUG 7113611.

Changes

Test waivers

Layer / File(s) Summary
Add test waivers
tests/integration/test_lists/waives.txt
The skip list adds entries for the dep16, helix8_moeep8, helix8_moetp2ep4, and tp1 configurations. Each entry references NVBUG 7113611.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Merge Risk: Minimal · up to 05786

This PR temporarily skips four SM107 Kimi K3 VL KDA metadata test cases that fail with an import error, and ties the skips to a tracked bug. No product behavior changes. Merge risk is minimal, and the waivers should be removed once the bug is fixed.

Architecture risk: Low · up to 05786

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: Added four SM107 Kimi K3 VL KDA metadata test entries to the skip list, covering the dep16, helix8_moeep8, helix8_moetp2ep4, and tp1 modeling configurations; all reference NVBUG 7113611.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title uses the required NVBugs format and the [infra] type, and it names the change: waiving four failed test cases. It does not describe the cause or that the waiver is for all platforms in a way…
Description check Passed The description covers the reason for the waiver, the failing tests, the bug link, and the removal condition. It follows the repository template loosely. It leaves the Test Coverage section with only …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title uses the required NVBugs format and the [infra] type, and it names the change: waiving four failed test cases. It does not describe the cause or that the waiver is for all platforms in a way that matches the scope exactly, but the main change is clearly stated. The title is fully related to the changeset.

Full details: Description check

Explanation

The description covers the reason for the waiver, the failing tests, the bug link, and the removal condition. It follows the repository template loosely. It leaves the Test Coverage section with only a review note and leaves the PR checklist unchecked. The description-to-template gap is small, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between f1856a4 and 05786bc.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the four lines against the CPU stage's inner pytest collection, the unittest waive matcher (tests/unittest/conftest.py), the waive pre-commit checks, and the failing pre-merge run L0_MergeRequest_PR #70908.

  • The node IDs match exactly.
  • Only the four failing parametrizations are skipped; the other tests in the file still run.
  • Because the case is cpu_only, the unscoped lines don't change any GPU stage.

Posted: 0 blockers, 0 possible blockers, 1 nice to have. The nice-to-have is about coordinating with the open fix #22545 and the sibling auto-waive PRs.

unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling] SKIP (https://nvbugs/7113611)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) The lines themselves look right. However, this break already has a fix open in #22545, and seven other auto-waive PRs cover other failures from the same break: #22523, #22524, #22534, #22536, #22539, #22543 and #22544, each under its own NVBug. The break comes from #22078 removing _fp4_mla_generation_num_blocks_device while #21996 still imports it at kimi_k3_vl/ops/attention/_k3_fp4_mla_cache.py:60.

#22545 links none of these bugs and doesn't touch waives.txt, and unwaiving here is a manual follow-up. Please link #22545 in nvbugs/7113611 and pick the merge order:

Otherwise this cpu_only check, which runs only in CPU pre-merge, stays skipped after the fix.

FYI, even all the waive PRs together won't make CPU pre-merge green. The kimi_k3_nvfp4__sm_107__{dep16,helix8_moeep8,tp1} directory entries fail at collection (L0_MergeRequest_PR #70908, CPU-Generic-arm-3: Interrupted: 1 error during collection), and a case-level waive can't reach that.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113611 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants