Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions tests/integration/test_lists/waives.txt
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,26 @@ unittest/_torch/attention/multi_gpu --ignore=unittest/_torch/attention/multi_gpu
unittest/_torch/attention/multi_gpu/test_mla_helix_context.py::test_k3_mla_helix_owned_prefill[tp1-cp4-gate-fp8-weights-bf16-kv] SKIP (https://nvbugs/7109810)
unittest/_torch/attention/sparse/csa2/test_module.py::test_fused_native_index_q_and_graph SKIP (https://nvbugs/7039506)
unittest/_torch/misc/test_guided_decoder_errors.py::test_executor_loop_fails_the_request_once_its_batch_is_synchronized SKIP (https://nvbugs/7068781)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[bf16-op_b2_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_table_b8_q1-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-128] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-cute_wide_verify_b2_q2-32] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-32] SKIP (https://nvbugs/7113764)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] (B3: CI breakage) test_context_and_decode_match_backend[fp8-op_b2_q1-128] is missing here, and it fails on DGX_B300 with the same ImportError. The test calls ops = _ops() at test_modeling_v2_k3_mla_attn_ops.py:415, before its off-SM107 pytest.skip at lines 419-425. _ops() goes through k3_mla_cache.py:40 to _k3_fp4_mla_cache.py:58-64 and raises ImportError: cannot import name '_fp4_mla_generation_num_blocks_device', so on SM103 this case errors out instead of skipping.

In L0 #70898 (PR #22261), DGX_B300-PyTorch-3 reports this file as tests=22, failures=21, skipped=1, and [fp8-op_b2_q1-128] is one of the 21. L0 #70869, #70899, #70903, #70904, #70907 and #70910 show the same result. With 20 of the 21 waived, DGX_B300-PyTorch-3 stays red for every PR (CI merges the TOT waive list into each run), so the 7 PRs listed in the description still fail when they re-run.

Suggested fix: add the case at its sorted position, right above this line:

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_context_and_decode_match_backend[fp8-op_b2_q1-128] SKIP (https://nvbugs/7113764)

and say 21 cases in the title and description. Alternatively, replace the 20 lines with one file-level entry, unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py SKIP (https://nvbugs/7113764). That works for three reasons:

  • on l0_b300.yml:221, the only list that runs this file, every case either fails with this ImportError or skips itself (test_nvfp4_generation_scatter_matches_backend is SM107-only);
  • the unittest waive matcher supports file-level entries (tests/unittest/conftest.py:346-361, precedent at waives.txt:207-208);
  • the un-waive becomes a single line.

unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-cached_kv] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[bf16_qkv-chunked_prefill] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-cached_kv] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_fp8_cached_prefix_context_matches_backend[fp8_qkv-chunked_prefill] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_context_matches_backend SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_nvfp4_split_k_context_append_and_prefix_gather_match_backend SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[bf16] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[fp8] SKIP (https://nvbugs/7113764)
unittest/_torch/modeling_v2/models/kimi_k3_vl/ops/attention/test_modeling_v2_k3_mla_attn_ops.py::test_state_matches_backend_roles[nvfp4] SKIP (https://nvbugs/7113764)
unittest/_torch/moe/test_dynamic_eplb_contracts.py::test_on_autotune_builds_balanced_powerlaw_input SKIP (https://nvbugs/7034332)
unittest/_torch/moe/test_moe_backend.py::test_fc1_prologue_wait_keeps_a_longer_stream_open SKIP (https://nvbugs/7099532)
unittest/_torch/moe/test_moe_backend.py::test_fc1_prologue_wait_preserves_exhaustion SKIP (https://nvbugs/7099532)
Expand Down
Loading