Skip to content

[https://nvbugs/7113413][infra] Waive 2 failed case(s) on all platforms - #22534

Closed
trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/be093f4ddfdbea0067bfd0165db3c592
Closed

trtllm-agent wants to merge 1 commit into
NVIDIA:mainfrom
trtllm-agent:auto-waive/be093f4ddfdbea0067bfd0165db3c592

Conversation

@trtllm-agent

@trtllm-agent trtllm-agent commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Temporarily waive 2 failed case(s) on all platforms with unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).

The native main-break detector (native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reports unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile broken on main for CPU: failure signature importerror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policy native_p3@1.0.0 selected it; https://nvbugs/7113413 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefix CPU), so every platform that runs the case fails it.

  • Bug: https://nvbugs/7113413 (details, logs and 7-day test history are in the bug)
  • Restore coverage when: Remove this line when https://nvbugs/7113413 is fixed or closed and the test passes post-merge; un-waive follows the bug, not the PR.
  • Scope reviewed by: auto-waive at 2026-11-18T06:36:08.494883+00:00

Tests in this break event:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile on CPU
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults on CPU

PRs whose pre-merge runs hit this break (rebase or re-run after this merges):

unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults SKIP (https://nvbugs/7113413)

What to check before merging:

  • The bug's test-history links show the cases failing on main, not only on the trigger PR.
  • The scope all platforms matches the failing stage and no narrower waiver already exists.
  • Nothing else is needed from you after merging: the bug owner removes these lines with the fix.
Provenance
  • Detector native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policy native_p3@1.0.0
  • Run b63a992e-b7c5-4815-92d3-fe7b0737fe16; snapshot SHA-256 5fb2f3fdddfe9ce39251cec27d66dc9e9c21975354c430fd53248a3ea4b859d8; config SHA-256 465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile: incident 21dd8cb16b1d12ad4b70cb84d224b8c6d59a1627db526a3b6f7626041d3e126e, observation 2a61b06647b52edeee9f938396358afe6bddfc180c8c08985cb87520a03bd5c9
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults: incident f2d0841006ea3e22b3119968980de2cefe16816da9bc56b64f7c4e1154b2fb6d, observation e4cfe3650cf109eb095ecb7983434721d8682f5ea9096a40ee3d7b228382f770

Test Coverage

Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.

PR Checklist

  • Please check this after reviewing the above items as appropriate for this PR.

Dev Engineer Review

  • The diff changes one file: tests/integration/test_lists/waives.txt, +2 lines, 0 removed. No source code, configuration, or API changes.
  • The two added lines are SKIP waivers for test_helix8_moetp2ep4_constants.py. Both cite NVBUG 7113413.
  • Risk: the waivers remove CPU coverage for these two tests on all platforms until the waivers are removed. The PR description states that the cause is a code-level import error on main. This change does not fix that error. It also does not show that main is fixed.
  • Scope: the waiver applies to every platform, not only the affected one. Reviewers should confirm this scope is intended.
  • The waivers should be removed after the bug is fixed or closed and the tests pass.

QA Engineer Review

Test files and test-list files changed:

  • Modified: tests/integration/test_lists/waives.txt (+2 entries, 0 removed).

Waivers added:

  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile (SKIP, https://nvbugs/7113413)
  • unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults (SKIP, https://nvbugs/7113413)

Coverage impact:

  • Both tests stop running in CI while the waiver is active. The tests check the Kimi K3 VL Helix8 MoE TP2/EP4 constants and the target class declarations.
  • No test function bodies changed. The waiver is the only change.
  • Test-list changes do not require test-db/ or qa/ list updates. This change does not add or move tests.

Coverage verdict: needs follow-up. The waiver removes coverage for the constants checks. Verify that the import error is fixed and that the waivers are removed after the tests pass.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt: Adds two SKIP entries for the Kimi K3 VL Helix8 constants tests under NVBUG 7113413. QA should confirm the tests pass on main after the underlying import error is fixed, then remove these entries.

Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
@tensorrt-cicd
tensorrt-cicd marked this pull request as ready for review November 18, 2026 06:41
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

/bot skip --comment "auto-waive: waive-only change"

@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The integration test skip list adds two Kimi K3 VL Helix8 MoE TP2/EP4 constants test entries. Both entries reference NVBUG 7113413.

Changes

Kimi K3 VL test waivers

Layer / File(s) Summary
Add constants test waivers
tests/integration/test_lists/waives.txt
Adds entries for the reference-profile constants test and the target-class constants declaration test. Both entries reference NVBUG 7113413.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Change: Other

Suggested reviewers: brnguyen2

Merge Risk: Minimal · up to af236

This change temporarily skips two failing constants tests for the Kimi K3 VL model, tracked by a bug. It does not change production behavior. The merge risk is minimal; the entries should be removed once the bug is fixed.

Architecture risk: Low · up to af236

The change affects 1 system.

Changed systems: tests

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — tests (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in tests/integration/test_lists/waives.txt: The skip list adds the Kimi K3 VL Helix8 MoE TP2/EP4 reference-profile constants test and target-class constants declaration test, both linked to NVBUG 7113413.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title follows the repository template with an NVBugs ID and the [infra] type. It names the main change, a waiver of two failed test cases on all platforms. The title is specific to the waives.txt …
Description check Passed The description covers the template's required content. It explains the waiver, the failing tests, the bug link, and the removal condition. The Description section is filled in. The Test Coverage sect…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title follows the repository template with an NVBugs ID and the [infra] type. It names the main change, a waiver of two failed test cases on all platforms. The title is specific to the waives.txt change, so it passes.

Full details: Description check

Explanation

The description covers the template's required content. It explains the waiver, the failing tests, the bug link, and the removal condition. The Description section is filled in. The Test Coverage section is thin and gives no specific test list, and the PR Checklist is unchecked. The description is mostly complete, so the check passes.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
Fix failing CI checks

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between befcc89 and af23602.

Files selected for processing: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 7 remain after this review.

@trtllm-agent-review trtllm-agent-review left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Posted 2 inline comments, both Nice to have; no blockers or possible blockers. The two waived ids are correct and the waive takes effect: both tests fail on the CPU stage when importing the target's modeling (ImportError from _k3_fp4_mla_cache.py:60, L0_MergeRequest_PR #70923). The file's third test passes and stays enabled. The lines are sorted and cite the bug filed for the failure.

unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile SKIP (https://nvbugs/7113413)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R2: Test lists and waives) These two lines clear only the kimi_k3_nvfp4__sm_107__helix8_moetp2ep4 entry. The same ImportError (cannot import name '_fp4_mla_generation_num_blocks_device', raised from kimi_k3_vl/ops/attention/_k3_fp4_mla_cache.py:60) stops the three sibling entries in l0_cpu.yml at collection: ...__sm_107__dep16 (169), ...__helix8_moeep8 (170) and ...__tp1 (172). Each of these directories has a test that imports its target's modeling at module level (test_dep16_k3_mla_ctx_split_k.py:14, test_helix8_k3_mla_ctx_split_k.py:14, tp1/test_k3_mla_ctx_split_k.py:20).

In L0_MergeRequest_PR #70923 (CPU-Generic-x86/arm shards 2-4), all three end with "Interrupted: 1 error during collection", so their copies of these two constants tests never run. Per-test lines can't fix that, because apply_waives_ut in tests/unittest/conftest.py only marks collected items. None of the other open auto-waive PRs covers these entries either. Until #22545 lands, the CPU stage stays red for the PRs the description asks to rebase or re-run.

If the waive route is kept, waive them as whole entries, in the same form as the existing full:sm100/unittest/bindings SKIP:

unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__dep16 SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moeep8 SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__tp1 SKIP (https://nvbugs/7113413)

This also covers the same entries on the GR100 post-merge list (l0_gr100.yml:152-154).

unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults SKIP (https://nvbugs/7113413)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R1: PR process) The fix for this ImportError is already open as #22545. It defines _fp4_mla_generation_num_blocks_device in the K3 family's _k3_fp4_mla_cache.py, and its description lists the target constants tests among those it fixes. But #22545 cites none of the auto-waive bugs and removes no waives.txt line.

This PR is also one of ten open auto-waive PRs for the same ImportError, each with its own bug (#22523, #22524, #22525, #22536, #22539, #22542, #22543, #22544, #22548). Seven of them insert at this same spot, so each merge conflicts with the next.

If this merges first, nothing ties these lines to the fix, and the two tests stay skipped after #22545 lands. If #22545 merges first, this PR waives two passing tests. Please coordinate the merge order: close this PR (and its siblings) once #22545 lands, or have #22545 delete these lines and reference nvbugs/7113413 so the bug closes with the fix.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@trtllm-agent

Copy link
Copy Markdown
Collaborator Author

Closing: nvbugs/7113413 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants