Repository navigation
Conversation
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
Signed-off-by: Yanchao Lu <yanchaol@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Problem
B200 pre-merge tests pay for repeated compilation, autotuning and model warmup. Several expensive parameter combinations reach the same check. Some checks also miss the regression they describe: three DSA comparisons accepted non-finite output, and the DeepSeek V4 transfer harness called a generation update with no generation requests.
Solution
Keep checks for the distinct failure modes and remove 889 GPU case executions across the selected workloads. A separate, always-skipped test-db entry is also removed.
A 96-hour history-based projection estimates 158 fewer cumulative test minutes (9.35% of 1,687) per complete B200 set. Across eight matched builds, the estimated median savings are 157 cumulative stage minutes and 5.67 allocated GPU-hours. The modeled slowest B200 shard improves by about 17 minutes when new directory keys use initial weights and 20 minutes once their durations are learned. This replay applies the incremental shard-planning effect on top of the prior per-build estimates. It credits only the reproducible 16-second capture-time change for the eight-GPU graph. Production CI latency and failure-rate changes require post-rollout measurements.
The following choices change pre-merge coverage.
--ignoreentries preserve excluded CPU-only siblings and the separate KDA entries. Editing a grouped file now selects the whole batch.Other changes in this PR and their coverage tradeoffs
0 → prompt_len. All 8 regular topologies, 13 uneven pipeline geometries, both layouts/orders, and 6 host checks remain. Keep the full layout/order cross on the sharded TP2/PP2 control, and both orders on selected asymmetric and sharded uneven geometries. The remaining repeated intersections are removed. CSA2 also opts into this real context update.CUTEDSLkeyword selector already runs the sixMEGAMOE_CUTEDSLnodes, so drop the separate invocation. Batch equivalent reference gathers/comparisons and limit the B200 MoE parent to two workers to avoid OOM retries.CBTS respects long pytest options, negative
-kexpressions and ignored children when selecting directory entries. Scoped agent guidance records how to evaluate regression coverage, real worker counts, cold/warm setup and native ownership.Test Coverage
Validation
Reviewed head
eb07a9adb933417fc558ad065277d991613171d3. Baseline9e7d33213e7be1809c8fad21e139da577af2bc0a. B200 comparisons below use actual pytest process time including startup and reporting. Time spent allocating nodes and preparing containers is outside that measurement.Earlier kernel, accuracy and video comparisons
Accuracy, refit, lifecycle and wrapper controls
The full CBTS Linux suite passed 1,089 checks on the CBTS revision retained in this head. New grouped selectors were also checked in the generated filtered test-db.
trt-test-dbrenders 426 single-GPU, 61 four-GPU and 55 eight-GPU PyTorch outer entries, versus 449/64/56 at baseline. Two-GPU/Ray/CPP stay at 33/16/2. The full PR takes the conservative CBTS fallback for the image builder. Mandatory pre-commit and DCO checks passed.Experiments used cached build
a13866f077dc83bf7704f69b555a55258ac614b6with relevant source comparisons and test overlays. Complete PR-head CI, rebuilt-image benchmarks and non-B200 hardware runs are outside this validation, including the shared B100 list condition and GB300 dp4 graph variant.PR Checklist