Skip to content

[None][test] Reduce B200 L0 setup, batch GPU unit tests and prune redundant cases - #22347

Closed
chzblych wants to merge 11 commits into
NVIDIA:mainfrom
chzblych:test/prune-dgx-b200-l0
Closed

chzblych wants to merge 11 commits into
NVIDIA:mainfrom
chzblych:test/prune-dgx-b200-l0

Conversation

@chzblych

@chzblych chzblych commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Problem

B200 pre-merge tests pay for repeated compilation, autotuning and model warmup. Several expensive parameter combinations reach the same check. Some checks also miss the regression they describe: three DSA comparisons accepted non-finite output, and the DeepSeek V4 transfer harness called a generation update with no generation requests.

Solution

Keep checks for the distinct failure modes and remove 889 GPU case executions across the selected workloads. A separate, always-skipped test-db entry is also removed.

A 96-hour history-based projection estimates 158 fewer cumulative test minutes (9.35% of 1,687) per complete B200 set. Across eight matched builds, the estimated median savings are 157 cumulative stage minutes and 5.67 allocated GPU-hours. The modeled slowest B200 shard improves by about 17 minutes when new directory keys use initial weights and 20 minutes once their durations are learned. This replay applies the incremental shard-planning effect on top of the prior per-build estimates. It credits only the reproducible 16-second capture-time change for the eight-GPU graph. Production CI latency and failure-rate changes require post-rollout measurements.

The following choices change pre-merge coverage.

Area Regression protection and tradeoff
Single-GPU unit registrations Group 22 small KDA, Kimi AttnRes, VSA and SOL file entries into four serial pytest runs. The three longest KDA files keep independent scheduling. Explicit --ignore entries preserve excluded CPU-only siblings and the separate KDA entries. Editing a grouped file now selects the whole batch.
CSA2 and native donor retirement The 14 shared-owner CSA2 cases now assert real source-history advancement. The prefix matrix keeps all 11 policy/remote-tail states and both asymmetric directions for selected hit, partial-eviction and no-reuse cases. One NIXL donor failure now checks the precise exception, cache close, GPU slot reclamation, retired-level residency and successful new requests in sequence.
Dummy-load smoke and mixed CUDA graphs DeepSeekV3Lite and GPT-OSS dummy weights now generate four tokens from a multi-page prompt, with capacity for a single request. Their old one-sample GSM8K harness allowed a score of zero. DeepSeek V4 Flash retains its two-request mixed-step greedy token comparison, with decode batch sizes 1/2 and prefill capture buckets 128/256. Larger unused graph capacities are relinquished.
Other changes in this PR and their coverage tradeoffs
Area Regression protection and tradeoff
Qwen3.6 DFlash / Qwen3.5 MTP Run Qwen3.6 DFlash VANILLA and the basic Qwen3.5 MTP control with untuned kernels. DFlash TRTLLM and the Qwen3.5 ring-replay/ADP controls retain tuning. Both controls run all 1,319 GSM8K samples with their original thresholds.
Cosmos3, LTX-2 and Wan CLI Cosmos3 Nano/Super and Wan T2V/I2V exercise the real CLI at four denoising steps with full output dimensions. This drops the longer default trajectory in those smokes. LTX-2 skips synthetic warmups for its one real request, retaining the 40-step generation, refinement stage and audio/video decode. Its CLI smoke consequently gives up the startup T2V/I2V warmup combination. Example defaults and numerical quality suites retain their settings.
DeepSeek V4 KV transfer Advance the source context history at the chosen side of a real NIXL transfer and check 0 → prompt_len. All 8 regular topologies, 13 uneven pipeline geometries, both layouts/orders, and 6 host checks remain. Keep the full layout/order cross on the sharded TP2/PP2 control, and both orders on selected asymmetric and sharded uneven geometries. The remaining repeated intersections are removed. CSA2 also opts into this real context update.
FlashInfer, sampler and MoE Bake an optional version/CUDA-matched SM100 kernel cache into the test image. The symbol-collision check loads the native runner and executes the operation. Runtime JIT remains available on cache miss. Remove 64 unused sampler-axis repetitions and two parent-covered MoE cases. The four-GPU CUTEDSL keyword selector already runs the six MEGAMOE_CUTEDSL nodes, so drop the separate invocation. Batch equivalent reference gathers/comparisons and limit the B200 MoE parent to two workers to avoid OOM retries.
Accuracy registrations and watermark MiniMax-M3 four-GPU selection 7 → 5 drops ADP/NVFP4/InferenceX and non-Eagle fused piecewise repetitions. Keep both KV dtypes, ADP, plain piecewise, and both Eagle3 projection paths. R1 keeps batch cap 32 over the otherwise identical batch-8 MTP control. DeepSeekV3Lite retains V1 MTP, V2 baseline/chunked MTP and explicit cache-reuse checks. Watermark keeps the dense model's full sampler assertions and drops the MoE model's distribution check.
Mamba and DSA Mamba replay 761 → 545, retaining all state dtypes, tile sizes 16/32/64, the 144-case single-step head-tile matrix and representative multistep trajectories. The main grid drops T=10/32 specializations. Thin intermediate DSA runtime shapes while retaining all 27 FP4 and 8 FP8 static variants. Require finite outputs within the valid context.
LTX2/FLUX unit tests, Ray refit and native lifecycle Share checkpoint loads for layer numerics and the original storage ratio >1.8. Keep dense/MoE × FP8/NVFP4 × full/partial refit checks, with two 1,024-token controls and shorter 128-token probes elsewhere. Lifecycle 11 → 8 retains actual NIXL/allocator ownership and both fatal directions. CPU retirement tests cover additional deadline/status combinations.

CBTS respects long pytest options, negative -k expressions and ignored children when selecting directory entries. Scoped agent guidance records how to evaluate regression coverage, real worker counts, cold/warm setup and native ownership.

Test Coverage

Validation

Reviewed head eb07a9adb933417fc558ad065277d991613171d3. Baseline 9e7d33213e7be1809c8fad21e139da577af2bc0a. B200 comparisons below use actual pytest process time including startup and reporting. Time spent allocating nodes and preparing containers is outside that measurement.

Workload Before → after Validation
Four B200 directory batches 1061.82 → 595.98 s (43.9%) in cumulative worker-process time 489 passed + the same 2 skips on first attempts. All 57 file/function/ignore/helper CBTS probes retained expected stages and split counts. New directory keys need new duration and coverage history.
CSA2 transfer, complete file 47 → 35 passed, 430.94 → 324.12 s (24.8%) The old generation update failed the new source-history control with actual 0, expected 65.
NIXL donor failure, affected cases 103.59 → 62.60 s Original 2 positive + 2 existing Fabric skips, final 1 positive + 1 existing Fabric skip. Native checks observed the precise failure and full reclamation of both GPU pools before a fresh request succeeded.
DeepSeekV3Lite / GPT-OSS dummy load 249.39 → 156.88 s / 116.61 → 101.05 s Four fresh AOT containers per model (ABBA). Each candidate processed 145 input tokens and produced four decode tokens. Negative checker controls rejected incomplete output. The shorter workload gives up running the dataset evaluator and default serving capacity against dummy weights.
DeepSeek V4 Flash mixed graph, eight GPUs 763.31 → 650.21 s in an isolated test-list pair No instrumentation in that pair. A separate four-run cold trace observed token parity and real mixed-step bucket 256 replay on all eight ranks. Capture time fell about 16 s across runs. Host startup and attention JIT also varied.
Earlier kernel, accuracy and video comparisons
Workload Before → after Validation
Qwen3.6 DFlash VANILLA, fresh AOT containers, ABBA 882.61 / 840.92 → 292.46 / 293.08 s (66.0%) All four passed. Accuracy 95.186 / 95.110 against 91.797. Temporary instrumentation verified the configured tuner, over 186k accepted draft tokens in each run and acceptance lengths 5.290 / 5.287.
Cosmos3 Nano / Super CLI, AOT, candidate first 228.81 → 177.17 s (22.6%) / 594.89 → 239.80 s (59.7%) Both sides decoded 189 frames at 1280×720. Independent original-first pairs without the optional cache also passed, 257.12 → 134.38 s and 573.42 → 209.94 s.
LTX-2 CLI, AOT, candidate first 347.96 → 195.97 s (43.7%) Both real two-stage requests completed 40 steps and produced 121 frames at 768×512, 24 fps and 16 kHz stereo audio.
DeepSeek V4 native KV transfer, complete file 749.46 → 485.52 s (35.2%), 64 → 40 passed Removed cases accounted for 255.76 s of baseline testcase time. Reinstating the old update failed on the first request with source history 0, expected 65.
Four-GPU configurable MoE Remove one 62.95 s process Across 19 original selectors, 1,391 selections covered 1,385 unique nodes. The retained broad selector returned 24 passed/54 skipped and the narrow selector 4 passed/2 skipped. All six outcomes and skip reasons matched.
FlashInfer native symbol test 1125.16 → 32.99 s Same operation passed with JIT disabled, zero builds. CPU cache install 41.24 s. Full-image export time was not measured.
Qwen3.5 MTP, both with AOT 530.40 → 233.47 s Accuracy 94.390 / 95.072 against 91.327. Acceptance length 3.502 / 3.509.
Wan T2V / I2V 687.77 → 440.75 s / 753.74 → 457.20 s Independent fresh containers. Verified both noise stages and decoded geometry/frame counts.
DSA FP4 / FP8, complete files 327.06 → 216.14 / 218.34 s, 105.26 → 100.20 / 100.02 s 1084 → 598 and 192 → 144 passed. Three all-NaN controls failed the new checks and passed the original ones.
LTX2 / FLUX, complete unit files 618.64 → 291.10 s / 601.30 → 319.05 s 39 → 32 and 27 → 16 passed. Final restored storage-ratio assertions passed seven targeted cases after those timings.
Mamba replay, fresh containers 487.04 → 429.63 s 761 → 545 passed.
Accuracy, refit, lifecycle and wrapper controls
  • MiniMax four-GPU targeted 5 → 3 passed, 2819.62 → 983.24 s. R1 eight-GPU targeted 2 → 1 passed, 820.99 → 363.54 s. The deleted cases consumed 1273.31 s and 248.72 s in the respective baselines. Retained cases benefited from warmup in both comparisons. The two other retained MiniMax InferenceX controls were outside that comparison.
  • DeepSeek BF16/reuse 7 → 4 passed with one existing waive on each side, 1776.01 → 959.54 s. Each positive evaluation used all 1,319 samples. Watermark 2 → 1 passed, retained case repeated and 45 separate checks passed. The deleted MoE case consumed 348.62 s. Its suite comparison had different cache conditions.
  • Lifecycle final 8 passed twice in 1101.99 / 573.06 s versus 1394.12 s for the original 11. Startup varied substantially. Ray final 8 passed in 1093.76 s on a fresh allocation. The full Ray baseline and final selection used separate allocations. The real cache-reset control passed.
  • Full MoE wrapper always returned 1492 passed and 577 existing skips. Warm baseline had 107 OOM retries versus zero with the candidate worker policy. A separate after/before/after run took 232.60 / 217.93 / 183.28 s with 0 / 63 / 0 OOM retries. All 28 injected MoE/sampler corruptions were rejected.

The full CBTS Linux suite passed 1,089 checks on the CBTS revision retained in this head. New grouped selectors were also checked in the generated filtered test-db. trt-test-db renders 426 single-GPU, 61 four-GPU and 55 eight-GPU PyTorch outer entries, versus 449/64/56 at baseline. Two-GPU/Ray/CPP stay at 33/16/2. The full PR takes the conservative CBTS fallback for the image builder. Mandatory pre-commit and DCO checks passed.

Experiments used cached build a13866f077dc83bf7704f69b555a55258ac614b6 with relevant source comparisons and test overlays. Complete PR-head CI, rebuilt-image benchmarks and non-B200 hardware runs are outside this validation, including the shared B100 list condition and GB300 dp4 graph variant.

PR Checklist

  • Retained checks and deliberate coverage reductions documented.
  • Direct B200 validation and its limitations reported.
  • Optional CI kernel cache and example CLI options documented in the change. No product-library API change.

Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
@chzblych chzblych changed the title [None][test] Reduce B200 MoE and sampler test overhead [None][test] Prune B200 L0 test matrices and repeated model loads Nov 16, 2026
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
@chzblych chzblych changed the title [None][test] Prune B200 L0 test matrices and repeated model loads [None][test] Reduce B200 L0 test matrices and cold-start cost Nov 17, 2026
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
@chzblych chzblych changed the title [None][test] Reduce B200 L0 test matrices and cold-start cost [None][test] Reduce B200 L0 model setup, test matrices and duplicate launches Nov 18, 2026
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
@chzblych chzblych changed the title [None][test] Reduce B200 L0 model setup, test matrices and duplicate launches [None][test] Reduce B200 L0 setup, batch GPU unit tests and prune redundant cases Nov 18, 2026
@chzblych chzblych closed this Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant