Repository navigation
[https://nvbugs/7113413][infra] Waive 2 failed case(s) on all platforms - #22534
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe integration test skip list adds two Kimi K3 VL Helix8 MoE TP2/EP4 constants test entries. Both entries reference NVBUG 7113413. ChangesKimi K3 VL test waivers
Estimated code review effort: 1 (Trivial) | ~2 minutes Change: Other Suggested reviewers: Merge Risk:
|
trtllm-agent-review
left a comment
There was a problem hiding this comment.
Posted 2 inline comments, both Nice to have; no blockers or possible blockers. The two waived ids are correct and the waive takes effect: both tests fail on the CPU stage when importing the target's modeling (ImportError from _k3_fp4_mla_cache.py:60, L0_MergeRequest_PR #70923). The file's third test passes and stays enabled. The lines are sorted and cite the bug filed for the failure.
| unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810) | ||
| unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506) | ||
| unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile SKIP (https://nvbugs/7113413) |
There was a problem hiding this comment.
[Nice to have] (R2: Test lists and waives) These two lines clear only the kimi_k3_nvfp4__sm_107__helix8_moetp2ep4 entry. The same ImportError (cannot import name '_fp4_mla_generation_num_blocks_device', raised from kimi_k3_vl/ops/attention/_k3_fp4_mla_cache.py:60) stops the three sibling entries in l0_cpu.yml at collection: ...__sm_107__dep16 (169), ...__helix8_moeep8 (170) and ...__tp1 (172). Each of these directories has a test that imports its target's modeling at module level (test_dep16_k3_mla_ctx_split_k.py:14, test_helix8_k3_mla_ctx_split_k.py:14, tp1/test_k3_mla_ctx_split_k.py:20).
In L0_MergeRequest_PR #70923 (CPU-Generic-x86/arm shards 2-4), all three end with "Interrupted: 1 error during collection", so their copies of these two constants tests never run. Per-test lines can't fix that, because apply_waives_ut in tests/unittest/conftest.py only marks collected items. None of the other open auto-waive PRs covers these entries either. Until #22545 lands, the CPU stage stays red for the PRs the description asks to rebase or re-run.
If the waive route is kept, waive them as whole entries, in the same form as the existing full:sm100/unittest/bindings SKIP:
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__dep16 SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moeep8 SKIP (https://nvbugs/7113413)
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__tp1 SKIP (https://nvbugs/7113413)
This also covers the same entries on the GR100 post-merge list (l0_gr100.yml:152-154).
| unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506) | ||
| unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile SKIP (https://nvbugs/7113413) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults SKIP (https://nvbugs/7113413) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) The fix for this ImportError is already open as #22545. It defines _fp4_mla_generation_num_blocks_device in the K3 family's _k3_fp4_mla_cache.py, and its description lists the target constants tests among those it fixes. But #22545 cites none of the auto-waive bugs and removes no waives.txt line.
This PR is also one of ten open auto-waive PRs for the same ImportError, each with its own bug (#22523, #22524, #22525, #22536, #22539, #22542, #22543, #22544, #22548). Seven of them insert at this same spot, so each merge conflicts with the next.
If this merges first, nothing ties these lines to the fix, and the two tests stay skipped after #22545 lands. If #22545 merges first, this PR waives two passing tests. Please coordinate the merge order: close this PR (and its siblings) once #22545 lands, or have #22545 delete these lines and reference nvbugs/7113413 so the bug closes with the fix.
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
Closing: nvbugs/7113413 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 2 failed case(s) on
all platformswith unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profilebroken on main for CPU: failure signatureimporterror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113413 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefixCPU), so every platform that runs the case fails it.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profileon CPUunittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaultson CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
all platformsmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.0b63a992e-b7c5-4815-92d3-fe7b0737fe16; snapshot SHA-2565fb2f3fdddfe9ce39251cec27d66dc9e9c21975354c430fd53248a3ea4b859d8; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile: incident21dd8cb16b1d12ad4b70cb84d224b8c6d59a1627db526a3b6f7626041d3e126e, observation2a61b06647b52edeee9f938396358afe6bddfc180c8c08985cb87520a03bd5c9unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults: incidentf2d0841006ea3e22b3119968980de2cefe16816da9bc56b64f7c4e1154b2fb6d, observatione4cfe3650cf109eb095ecb7983434721d8682f5ea9096a40ee3d7b228382f770Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt, +2 lines, 0 removed. No source code, configuration, or API changes.test_helix8_moetp2ep4_constants.py. Both cite NVBUG 7113413.QA Engineer Review
Test files and test-list files changed:
tests/integration/test_lists/waives.txt(+2 entries, 0 removed).Waivers added:
unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_constants_are_the_reference_profile(SKIP, https://nvbugs/7113413)unittest/_torch/modeling_v2/models/kimi_k3_vl/kimi_k3_nvfp4__sm_107__helix8_moetp2ep4/test_helix8_moetp2ep4_constants.py::test_target_class_declares_its_constants_and_no_model_defaults(SKIP, https://nvbugs/7113413)Coverage impact:
test-db/orqa/list updates. This change does not add or move tests.Coverage verdict: needs follow-up. The waiver removes coverage for the constants checks. Verify that the import error is fixed and that the waivers are removed after the tests pass.
Per-File QA Perspective
tests/integration/test_lists/waives.txt: Adds two SKIP entries for the Kimi K3 VL Helix8 constants tests under NVBUG 7113413. QA should confirm the tests pass on main after the underlying import error is fixed, then remove these entries.