Repository navigation
[https://nvbugs/7113611][infra] Waive 4 failed case(s) on all platforms - #22542
trtllm-agent wants to merge 1 commit into
Conversation
Signed-off-by: trtllm-agent <296075020+trtllm-agent@users.noreply.github.com>
|
/bot skip --comment "auto-waive: waive-only change" |
|
No actionable comments were generated in the recent review. WalkthroughThe skip list now includes four SM107 Kimi K3 VL KDA metadata test configurations. Each entry references NVBUG 7113611. ChangesTest waivers
Estimated code review effort: 1 (Trivial) | ~2 minutes Change: Other Merge Risk:
|
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
trtllm-agent-review
left a comment
There was a problem hiding this comment.
I checked the four lines against the CPU stage's inner pytest collection, the unittest waive matcher (tests/unittest/conftest.py), the waive pre-commit checks, and the failing pre-merge run L0_MergeRequest_PR #70908.
- The node IDs match exactly.
- Only the four failing parametrizations are skipped; the other tests in the file still run.
- Because the case is
cpu_only, the unscoped lines don't change any GPU stage.
Posted: 0 blockers, 0 possible blockers, 1 nice to have. The nice-to-have is about coordinating with the open fix #22545 and the sibling auto-waive PRs.
| unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810) | ||
| unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506) | ||
| unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781) | ||
| unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling] SKIP (https://nvbugs/7113611) |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) The lines themselves look right. However, this break already has a fix open in #22545, and seven other auto-waive PRs cover other failures from the same break: #22523, #22524, #22534, #22536, #22539, #22543 and #22544, each under its own NVBug. The break comes from #22078 removing _fp4_mla_generation_num_blocks_device while #21996 still imports it at kimi_k3_vl/ops/attention/_k3_fp4_mla_cache.py:60.
#22545 links none of these bugs and doesn't touch waives.txt, and unwaiving here is a manual follow-up. Please link #22545 in nvbugs/7113611 and pick the merge order:
- if [None][fix] Kimi K3 Modeling V2: follow the FP4 MLA backend API on main #22545 lands first, don't merge these lines;
- if this lands first, have [None][fix] Kimi K3 Modeling V2: follow the FP4 MLA backend API on main #22545 or a follow-up delete them along with the sibling waives.
Otherwise this cpu_only check, which runs only in CPU pre-merge, stays skipped after the fix.
FYI, even all the waive PRs together won't make CPU pre-merge green. The kimi_k3_nvfp4__sm_107__{dep16,helix8_moeep8,tp1} directory entries fail at collection (L0_MergeRequest_PR #70908, CPU-Generic-arm-3: Interrupted: 1 error during collection), and a case-level waive can't reach that.
|
Closing: nvbugs/7113611 is closed, so this waive is no longer needed. If the failure comes back, the bot files a new bug and a new PR. |
Description
Temporarily waive 4 failed case(s) on
all platformswith unscoped line(s): the failure is a code-level error that fails on every platform, first seen on CPU. Opened automatically by auto-waive (draft; a human reviews and merges).The native main-break detector (
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1) reportsunittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling]broken on main for CPU: failure signatureimporterror: cannot import name _fp#_mla_generati...llm/_torch/attention/backends/fp#_mla/metadata.py), onset 2026-11-18T02:11:36Z, latest failure 2026-11-18T04:58:01Z. Policynative_p3@1.0.0selected it; https://nvbugs/7113611 tracks the fix. The line is unscoped: the failure is a code-level error that does not depend on the GPU (seen on CPU; stage prefixCPU), so every platform that runs the case fails it.Tests in this break event:
unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling]on CPUunittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moeep8.modeling]on CPUunittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moetp2ep4.modeling]on CPUunittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__tp1.modeling]on CPUPRs whose pre-merge runs hit this break (rebase or re-run after this merges):
What to check before merging:
all platformsmatches the failing stage and no narrower waiver already exists.Provenance
native-p3/k3-w12h-fresh12h-nopass-hostclock/1.0.1, policynative_p3@1.0.07ad252a8-926c-434c-abaf-ecfc58766ef8; snapshot SHA-2563b90e77c8c2055a399eed345e70d86e1c23bb376540f1585c66c4d5cfbddac74; config SHA-256465a88e1b18cbe3245f999c7a82690481adb9fa9d03086d8e5e54ea20b3e7190unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__dep16.modeling]: incidente86058814659f0b540cce18fbe7b2c65647cf27d17a9d702a16fca0d500f1e9d, observation97143d1897944af407f39549aa82ca6e88f59f36b5afb31adf6793f4758b592funittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moeep8.modeling]: incidentea18b5709e89ca6bdbf5a769103ef8e167c571a4f47aff87ef2aa6eeb3b91623, observationf78e202d9d825600318aa21ba8f41e328c50eca83d63bdbf16907abf4ad41db1unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__helix8_moetp2ep4.modeling]: incidentcd50fa2c313670400c01e00087f5e3cde567eef43f92fbc9cdd51c887f142929, observation288e025ce777474ccf80bdedbb94e5bf87a0825fad9c6d5df7ebe90ec7f84cd6unittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py::test_sm107_glue_uses_the_family_metadata[tensorrt_llm._torch.modeling_v2.models.kimi_k3_vl.kimi_k3_nvfp4__sm_107__tp1.modeling]: incident59beeef11201710177b6a1e01af707292129846ee479583032f334dc0ee8854d, observationcddbbba3af55323fcd55a6a5a6340589a28c3301f6626297691adb8b6fb2b332Test Coverage
Review the incident evidence and selector scope before merging. This proposal does not
establish that main is fixed. Confirm restoration separately after the corrective change.
PR Checklist
Dev Engineer Review
tests/integration/test_lists/waives.txt: adds four SKIP waiver entries. Each entry targets atest_sm107_glue_uses_the_family_metadatacase inunittest/_torch/modeling_v2/models/kimi_k3_vl/common/test_k3_kda_metadata.py. The four parameter IDs arekimi_k3_nvfp4__sm_107__dep16,helix8_moeep8,helix8_moetp2ep4, andtp1. All four referencehttps://nvbugs/7113611.main. This PR does not establish thatmainis fixed.QA Engineer Review
tests/integration/test_lists/waives.txt(+4 lines, 0 removed).test_sm107_glue_uses_the_family_metadatawith the four parameter IDs listed above, all tied to NVBUG 7113611.Per-File QA Perspective
tests/integration/test_lists/waives.txt: adds four SKIP waivers for the SM107 KDA metadata cases on all platforms. A QA engineer should confirm the cases pass after the bug is fixed and then remove these entries.