Skip to content

[dev] Share CUDA graph slots across dynamic CP variants - #6479

Open
xiaoyao0115 wants to merge 7 commits into
NVIDIA:devfrom
xiaoyao0115:dcp-cg-slot-pool
Open

xiaoyao0115 wants to merge 7 commits into
NVIDIA:devfrom
xiaoyao0115:dcp-cg-slot-pool

Conversation

@xiaoyao0115

@xiaoyao0115 xiaoyao0115 commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Lets dynamic context parallelism (DCP) use layer-wise Transformer Engine CUDA graphs. Each CP size gets its own graph bank, and all banks share one TE memory pool. FP8 attention and fine-grained activation offloading also work inside these graphs. DSv4/MLA backward metadata moves into saved tensors, the only state that shared graph memory protects.

Depends on NVIDIA/TransformerEngine#3353 as a whole (head 15fe37e0), not only its make_graphed_callables(_graph_memory_slots=...) argument. These graphs also rely on its slot offload (_graph_memory_offload_context), on the FP8 attention scales and THD all-gather metadata it saves through autograd, and on its physical-buffer saves of quantized tensors under CPU offload v1. Megatron checks only for _graph_memory_slots (DCP + CUDA graphs raises at construction without it) and _graph_memory_offload_context (graph offload raises at capture without it). Runs without DCP graphs do not need the TE change. Packed Gated DeltaNet capture (Qwen3.5) is left to a follow-up PR.

Based on dev 43a2246f4.

Design

  • Banks. TECudaGraphHelper captures every power-of-two CP size from --min-dynamic-context-parallel-size to DP×CP with that size's runtime group ([DCP V2 1/4] Define the runtime dynamic-CP group contract #7045). At replay, a layer picks its bank from PackedSeqParams.local_cp_size and checks the microbatch's CP group against it.
  • Shared memory. One slot ring, sized from the liveness of every PP/VPP schedule the DCP scheduler can produce. Saved-tensor and gradient arenas are colored so live frames never share storage; variants of a slot share addresses, so memory scales with the largest variant, not the number of banks.
  • Capture state. THD capture seeds a bounded RoPE table and cu_seqlens per bank. While TE captures, each RotaryEmbedding keeps the tables it returns in a per-module dict until the graphs are deleted, so a table built in warmup is still there at stream capture; a miss during stream capture raises instead of allocating the table in the graph pool. Banks share the leading tensor of each capture input after checking that layouts and aliasing match. The DCP scheduler caps each output pack at thd_max_packed_sequences. The capture bound counts source-item subsamples separately (SFT: seq_length // pad_divisor + 1; varlen, mock and other sources: 1), and each step's microbatch count is checked against it. The slot liveness proof enumerates only the microbatch counts up to the schedule's warmup plus lcm(slots, VPP group size), because longer schedules repeat their live sets, so planning cost does not grow with that bound.
  • FP8 attention. DCP graphs accept fp8_dot_product_attention and fp8_multi_head_attention. TE#3353 saves the FP8 scale snapshots that backward reads, and the per-step THD all-gather metadata, through autograd, so the slot arenas protect them like any other saved tensor.
  • Activation offloading. Inside a captured scope, fine-grained offloading goes through TE's slot saved-tensor hook instead of Megatron's own hook. Offloaded saves go to pinned host arenas colored like the GPU saved arenas (CP banks share the largest one). Forward captures the D2H copy and backward the H2D copy on the offloader's streams, and saved views are rebuilt with their aliases. The policy is frozen per (module, scope) after eager warmup: a scope is offloaded only if every warmed instance was. Because D2H runs on its own stream, TE first copies each offloaded save on the compute stream, so a later in-place write cannot change what is copied. Quantized tensors are saved through their physical buffers, so the CPU copy keeps FP8 data and scales. Offload outside captured scopes and offload with non-DCP graphs are unchanged.
  • DSv4/MLA. CSA sparse attention, cuDNN sparse attention, the fused DSA indexer loss and fused MLA RoPE keep their backward metadata in save_for_backward instead of ctx attributes. Eager numerics are unchanged.

Changes for runs without DCP (shared paths):

  • Metadata-only THD padding on middle PP stages uses the CP-local length like the first stage.
  • The CP=1 dp_balanced scheduler gathers on the equivalent DP×CP group.
  • Fixed-CP THD capture seeds the full RoPE table and graph cu_seqlens.
  • Attention-only replay passes the real PackedSeqParams to the eager MLP.
  • PP stages without graphable layers join the slot-ring all-reduce.
  • delete_cuda_graphs() no longer asserts that graphs exist.
  • TE graph capture keeps its RoPE tables per module (above). Before, more than 32 distinct tables could evict one that a captured graph still reads.
  • THD RoPE whose frequency table is shorter than max_seqlen takes the unfused path. TE's fused THD RoPE reads the table up to max_seqlen.

Review follow-up

  • RoPE tables used during TE capture are owned per module until graph deletion; a DSA indexer graph replays correctly after more than 32 other tables evict its table from the shared cache.
  • THD all-gather CP keeps its per-step metadata in saved tensors (TE#3353) instead of rejecting cp_comm_type="all_gather".
  • The FP8 attention scale snapshot is a saved tensor (TE#3353) instead of a ctx attribute, so FP8 attention stays enabled.
  • Offload scopes inside the graph delegate to TE's slot hook instead of nesting Megatron's hook. Whole-layer capture with offloading is rejected under DCP.
  • Leading capture inputs are shared across CP banks.
  • The capture bound uses a source-item subsample bound instead of thd_max_packed_sequences. With PP, --dataloader-inter-document-masking is rejected because its items have no such bound. The thd_max_packed_sequences docstring describes the DCP cap.
  • The slot liveness proof stops at one schedule period past the warmup, so the larger SFT bound does not make capture planning quadratic in it.
  • A bit-exact determinism replay covers the short-table THD RoPE fallback.
  • Style: -> None on the capture wrapper, double quotes on added string literals.

Commits

  1. f8aa5d759 Share TE CUDA graph memory across dynamic-CP graph banks
  2. bace48590 Keep DSv4 and MLA RoPE backward metadata in saved tensors
  3. 8bd9e61e4 Fix dynamic-CP graph capture safety and offload compatibility (slot offload, per-module RoPE tables, SFT source bound, shared leading inputs)
  4. 8ecc7c0b6 Use double quotes for string literals added by dynamic-CP graphs
  5. f5fa8b50f Add a determinism replay for the short THD RoPE table fallback
  6. f689dd1f0 Reject dynamic-CP graph inputs without a schedule bound or offload boundary
  7. 9da6607dd Bound the dynamic-CP slot liveness proof by the schedule period

Results (earlier revision, needs rerun)

Measured on cdfa7c212 (dev a5aa637fa) with TE#3353 8980cd09. That run predates the rebase, FP8/offload support and the review fixes, and has not been rerun on 9da6607dd. FP8 attention and offloading have no end-to-end training numbers.

Qwen3-30B-A3B (48 layers, random init) on 8×GB200, TP1 PP4 VPP4 EP2, packed variable-length sequences, GBS 16, --deterministic-mode, 50 steps. All runs use the same initial weights and data.

Run Step time Peak reserved Loss vs. eager
eager DCP (CP 1/2) 4.56 s 92.8 GiB
DCP + CUDA graphs (attn moe_router moe_preprocess) 3.81 s (−16.5%) 102.5 GiB bit-identical
DCP + CUDA graphs (attn) 4.14 s (−9.3%) 94.0 GiB bit-identical
eager, fixed CP 2 4.96 s 92.8 GiB
fixed CP 2 + CUDA graphs 4.15 s (−16.3%) 100.8 GiB bit-identical

Qwen3-30B-A3B LM loss, eager vs CUDA graphs (cdfa7c212)

Bit-identical: LM loss, grad norm and MoE auxiliary loss at every step, and the final weights. Both CP banks are replayed (28 of 50 steps mix CP 1 and CP 2).

Tests

  • tests/unit_tests/transformer/test_thd_cuda_graph.py: slot plan and liveness colors (property test over PP 2/4/8 × VPP 1/2/4 with VPP group size PP or 2×PP: slots and colors are proved over the truncated counts as in capture, then every reachable count up to twice the limit is replayed with an independent overlap check), capture bounds against scheduler output, bank contexts, capture retry and delete, THD RoPE bounds, leading-input reuse across banks, FP8/FP4 config acceptance.
  • tests/unit_tests/transformer/test_transformer_layer.py, test_mhc_attention_graph_split.py, test_cuda_graphs.py, test_vision_cuda_graphs.py: bank selection at replay, mHC attention split, slot-aliased variant order, delete before create.
  • tests/unit_tests/transformer/test_hybrid_dcp_cuda_graph.py: hybrid mHC + MoE replay vs. eager across banks.
  • tests/unit_tests/transformer/test_dcp_cuda_graph_offloading.py (new): offload event boundary, frozen scope policy, and a 4-rank PP2×DP2 run that compares losses and parameter gradients with eager for BF16/FP8 × dense/MoE × PP2/PP2-VPP2. It uses CP 1/2 banks, physical-slot wrap, microbatch counts 12 → 8 → 12 and core_attn/attn_proj offload.
  • tests/unit_tests/transformer/test_rope.py: RoPE and YaRN tables beyond the 32-entry cache survive capture.
  • tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_dsa.py: a DSAIndexer captured through TECudaGraphHelper replays like eager after 40 other tables evict its RoPE table from the shared cache and the freed memory is overwritten.
  • tests/unit_tests/test_sequence_packing.py, tests/unit_tests/training/test_train_step_schedule_plumbing.py: microbatch-count check, DP×CP gather group, sample regrouping, train() helper construction from parsed defaults, SFT source-subsample bound.
  • tests/unit_tests/transformer/test_transformer_config.py, tests/unit_tests/test_hybrid_context_parallel_args.py: dynamic-CP config and argument rules, including whole-layer capture + offload and PP + inter-document masking.
  • tests/unit_tests/fusions/test_fused_mrope.py, tests/unit_tests/fusions/test_mla_yarn_rope_apply.py, tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_csa.py, test_csa_fused_sparse_attention_dev.py, test_attention_variant_dsa.py: short THD RoPE fallback, DSv4/CSA/DSA/MLA saved-tensor paths, rejection of local graphs for packed DSA under DCP.
  • tests/unit_tests/determinism/kernels/test_te_wrappers.py (+ manifest.py): bit-exact forward/backward replay of the short-table THD RoPE fallback for CP1 and both CP2 ranks.

TE-side tests (tests/pytorch/test_cuda_graphs.py, tests/pytorch/attention/test_saved_quantizer_scales.py) are in NVIDIA/TransformerEngine#3353.

Validation on 9da6607dd + TE#3353 15fe37e0

NGC PyTorch 26.03 containers with TE#3353 built from 15fe37e0.

Suite 8×H100 4×GB200
DCP graph unit tests: test_thd_cuda_graph.py, test_rope.py, test_hybrid_dcp_cuda_graph.py, test_mhc_attention_graph_split.py, test_pos_embedding_cuda_graph.py, test_hybrid_context_parallel_args.py, test_train_step_schedule_plumbing.py (4 ranks, one process per rank) 273 passed, 2 skipped per rank 273 passed, 2 skipped per rank
test_dcp_cuda_graph_offloading.py PP2×DP2 matrix and offload tests (4 ranks) 9 passed, 4 skipped per rank (FP8 needs SM100) 13 passed per rank
test_attention_variant_dsa.py::TestDSAIndexer::test_graph_replay_keeps_rope_table_beyond_lru_capacity 2 passed 2 passed
Same test with both RoPE table owners disabled fails (illegal address on replay) fails (illegal address on replay)
test_te_wrappers.py -k rope, test_manifest.py 351 passed, 1 skipped 352 passed
test_transformer_config.py -k dynamic_cp / test_sequence_packing.py (8 ranks) 6 passed / 73 passed per rank 6 passed / –
CI unit-test buckets (transformer, moe, experimental attention, models, ssm, fusions, pipeline, a2a, data, determinism, training), compared test by test with dev 43a2246f4 no failure that dev does not also have same, for the launch_on_gb200 tests

Limitations

  • CP sizes and DP×CP must be powers of two.
  • Dynamic-CP graphs need --cuda-graph-impl transformer_engine with cuda_graph_dynamic_microbatches. full_iteration/local graphs (including dev's experimental packed-DSA path), step_batch_size_schedule, delayed wgrad, EP overlap, moe_paged_stash and mamba scopes are rejected.
  • With PP, --dataloader-inter-document-masking is rejected for non-SFT data. Other non-SFT sources are assumed to carry one sequence per item; a step outside the captured range fails the runtime check. The SFT bound is conservative; a larger bound only widens the runtime range, not the capture-time liveness proof.
  • Megatron's FP8 attention graph tests use DelayedScaling (fp8_recipe="delayed"); the TE tests also cover current scaling. MXFP8/FP4 attention passes config validation without numerical coverage here. FP8 THD tests need SM100+ and cuDNN ≥ 9.23, so they skip on Hopper. Megatron's DCP graph replay tests do not use cp_comm_type="all_gather"; all-gather replay is covered only by the TE tests.
  • Offloading inside DCP graphs needs partial cuda_graph_modules scopes; whole-layer capture with offloading is rejected. The frozen per-scope policy can offload less than eager. There is no ahead-of-use prefetch: H2D is joined before backward uses the tensor. Async D2H adds a transient compute-stream copy of each offloaded save within the forward graph. TE allocates the host arenas outside fine_grained_offloading_buffer_size_gib. Tested with GPT attention scopes only; MLA and hybrid offload under DCP graphs are untested. NVTE_CPU_OFFLOAD_V1=1 is required, as for fine-grained offloading in general.
  • CI skips the TE-slot tests until [PyTorch] Share CUDA graph memory across dynamic CP variants TransformerEngine#3353 is in the pinned TE. The offload pipeline test needs WORLD_SIZE=4 (FP8 cases also need SM100): the 8-rank H100 unit-test jobs skip it, and the GB200 jobs run only launch_on_gb200 tests.

Determinism

The determinism-exempt label covers five registered kernels that the coverage check still lists: fused MLA YaRN RoPE, attention dispatch, DSv4 q RMSNorm, DSA dispatch and MoE experts. Their edits move backward metadata into save_for_backward or pass the owning module to the offload interface; they do not change arithmetic or kernel selection. The fused DSA indexer-loss backward is not run by the registered DSA determinism tests; test_attention_variant_dsa.py::TestFusedDSAIndexerLossGradient checks it against autograd. The one dispatch change that does (the short-table THD RoPE fallback) has its own bit-exact replay in test_te_wrappers.py.

Issue tracking

Linked issue: none.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

🤖 Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test 2a1d969

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test a24ea4d

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test 7e6ba4c

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test 8bb6345

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test cdfa7c2

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed the light review across all changed files and their directly affected paths. One scheduling-bound correctness issue, one determinism-coverage gap, and two unenforced style-guide violations remain. Tests were inspected, not executed; Transformer Engine’s external slot allocator was not available for inspection.

[P2] Add bit-exact coverage for the new THD RoPE dispatch branch

Failure — coverage: The new short-frequency-table check changes dispatch from TE to the unfused implementation. The added fusion tests check forward closeness and graph replay, but not bit-exact forward/backward repeatability. No determinism test changes accompany this dispatch change, despite the required kernel-testing contract. The dispatch is already registered against the TE wrapper tests.

Correction: Add a short-table forward/backward replay case to the registered test module. Suggested addition, not executed; verify before adopting:

@pytest.mark.parametrize("cp_size, cp_rank", [(1, 0), (2, 0), (2, 1)])
def test_thd_short_rope_replays_fwd_bwd(monkeypatch, cp_size, cp_rank):
    from megatron.core.models.common.embeddings import rope_utils

    seeded()
    config = _config(apply_rope_fusion=True)
    group = SimpleNamespace(size=lambda: cp_size, rank=lambda: cp_rank)
    cu = torch.tensor([0, 4, 8], dtype=torch.int32, device="cuda")
    freqs = torch.randn(4, 1, 1, 16, device="cuda")
    t = torch.randn(
        8 // cp_size, 3, 20, device="cuda",
        dtype=torch.bfloat16, requires_grad=True,
    )

    def reject_te(*args, **kwargs):
        raise AssertionError("short THD tables must bypass TE")

    monkeypatch.setattr(rope_utils, "fused_apply_rotary_pos_emb_thd", reject_te)

    def apply(t):
        return rope_utils.apply_rotary_pos_emb(
            t, freqs, config, cu_seqlens=cu, cp_group=group, max_seqlen=8
        )

    assert_replays_bit_exact(
        apply, (t,), replays=3, backward=True, contention=True,
        what=f"short THD RoPE CP{cp_size}/rank{cp_rank}",
    )

Preservation: Retain the existing full-table SBHD/THD fused-RoPE replay cases. The short materialized-table fallback must not redirect raw three-axis mRoPE or change their existing fusion/parity coverage.

Acceptance — proposed: For all three CP cases above, TE must not be called, and every output and input gradient must be byte-identical across replays. Existing full-table fused cases and the new forward/graph-parity tests must continue passing.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread megatron/training/training.py Outdated
# datasets yield pre-packed items of up to thd_max_packed_sequences sequences.
max_subsamples_per_item = 1
if not args.use_varlen_dataset and args.thd_max_packed_sequences is not None:
max_subsamples_per_item = args.thd_max_packed_sequences

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Bound source subsamples independently of the destination pack capacity

Failure: thd_max_packed_sequences limits each repacked output, not the number of sequences in a source dataset item. SFTDataset packs conversations up to sequence_length without consulting this setting, and the scheduler unpacks every nonempty interval.

A concrete supported case is TP1, PP2, static CP1, DP2, GBS2, seq_length=256, local token capacity 64, and thd_max_packed_sequences=2 with dummy-tail padding. Each of the two source items can contain intervals [80, 80, 96]. Every interval needs CP2; the one-real-sequence output cap therefore produces six microbatches. This assignment instead declares two subsamples per item, so the capture bound is four. After capture, runtime validation rejects this valid step instead of training it.

Correction: Change _get_dynamic_cp_cuda_graph_helper_kwargs to pass an independently justified source-item subsample bound. For in-tree SFT, derive it from source pack length and the dataset’s CP/SP padding granularity, or expose the dataset’s actual maximum through the helper plumbing. Do not use the post-repacking capacity as that bound, and retain the runtime liveness guard. The tight implementation depends on how the source-dataset bound is exposed, so substituting another unexplained constant would not be a complete fix.

Preservation: Varlen items must still contribute exactly one sequence to this calculation. Fixed-CP/non-THD helper construction must still work with thd_max_packed_sequences=None. Destination packs must retain their one-real-sequence-plus-dummy limit in this example.

Acceptance — proposed, not executed: Feed six lengths [80,80,96,80,80,96] to DefaultDynamicCPScheduler with two total DCP ranks, local capacity 64 and max_num_seqs=1; assert six output microbatches. Build the capture contract from the corresponding two-item source setup and assert its bound covers six and runtime validation accepts six. Retain rejection above the proven bound and the parsed-default/varlen assertions in the plumbing tests, updating the non-varlen expectation to the independently derived source bound.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 6bbffab. The capture bound no longer uses thd_max_packed_sequences. For --sft data it is seq_length // d + 1 per source item, where d is the CP/SP padding divisor; other data keeps 1. Pre-packed data with --dataloader-inter-document-masking has no usable bound, so argument validation now rejects it when dynamic-CP graphs run with PP>1 (6b43bf5). test_sft_source_subsample_bound_covers_dynamic_cp_repacking runs your example and gets six microbatches, all covered by the bound.

class _DynamicCPCaptureCallable(torch.nn.Module):
"""Set one dynamic-CP context immediately before capturing a wrapped layer."""

def __init__(self, module, context_setter, context):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Annotate the new capture-wrapper constructor

Failure — static: The new constructor lacks the -> None annotation required by the protected style guide. This is a source-style violation, not a runtime failure.

Correction: Replace only the signature:

Suggested change
def __init__(self, module, context_setter, context):
def __init__(self, module, context_setter, context) -> None:

Preservation: Leave module registration, grouped-prefix exposure, capture context and training state unchanged.

Acceptance — static/proposed: Confirm the constructor is annotated and its body is unchanged. Retain the grouped-prefix test’s assertions that model state-dict keys do not change and the temporary callable exposes the prefix’s parameters and buffer. The suggestion was not executed; verify it before applying.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6bbffab: _DynamicCPCaptureCallable.__init__ is annotated -> None.

Comment thread megatron/core/datasets/data_schedule.py Outdated

def _validate_cuda_graph_microbatch_count(config, num_microbatches: int) -> None:
"""Require runtime packing to stay within the captured schedule proof."""
limit = getattr(config, '_cuda_graph_num_microbatches', None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Use double quotes for newly introduced string literals

Failure — static: The new helper uses single-quoted attribute names, contrary to the style guide’s double-quote rule. The same issue occurs in newly added capture-state literals in cuda_graphs and the RoPE-bound helper. Black explicitly preserves quote style, so formatting does not repair this convention.

Correction: Normalize newly added literals to double quotes without restyling untouched code. For this line:

Suggested change
limit = getattr(config, '_cuda_graph_num_microbatches', None)
limit = getattr(config, "_cuda_graph_num_microbatches", None)

Apply the same delimiter-only correction to the other newly introduced literals.

Preservation: Keep string values, attribute names, dictionary keys and error messages identical; this must not alter capture-state lookup or validation behavior.

Acceptance — static: Inspect the added lines for double-quoted literals and verify their decoded values are unchanged. No runtime behavior change or new runtime test is required. Verify the suggestion before applying.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. 6bbffab converts the literals you linked, and 77c1d98 converts the remaining single-quoted literals on lines this PR adds.

@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

/ok to test 52953e2

Comment on lines +268 to +278
def _retain_cuda_graph_rope_tensors(self, *rope_tensors: Optional[torch.Tensor]) -> None:
"""Retain MLA-internal RoPE allocations used by TE CUDA graphs."""
if getattr(self.config, 'cuda_graph_impl', 'none') != 'transformer_engine':
return

from megatron.core.transformer.cuda_graphs import is_graph_capturing

if not is_graph_capturing():
return
refs = self.__dict__.setdefault('_cuda_graph_static_tensor_refs', {})
refs.update((id(tensor), tensor) for tensor in rope_tensors if tensor is not None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[CRITICAL Memory safety] This pins RoPE tables only for MLASelfAttention. Other in-graph producers read the table that RotaryEmbedding/YarnRotaryEmbedding.forward return, and its only owner is a class-wide @lru_cache(maxsize=32) (rotary_pos_embedding.py:181, yarn_rotary_pos_embedding.py:159). These producers are:

  • DSAIndexer.forward_before_topk with the default dsa_indexer_rope_interleaved=False (dsa.py:2148-2151)
  • with fusion off: AbsorbedMLA (absorbed_mla.py:546-548), DSv4 and CSA

Under DCP every bank adds one key per layer (min(max_seqlen_per_dp_cp_rank*cp, sample bound)). TE warms the largest CP first (TE transformer_engine/pytorch/graph.py:935-939 @ #3353), but the smallest CP is captured first (cuda_graphs.py:4112, 4350).

Trigger: DSA, 16 layers per stage, DPxCP=8, and seq_length >= 8 * max_seqlen_per_dp_cp_rank, so every bank keeps its own table length and there are 64 keys. In general it fires once layers per stage × distinct per-bank table lengths exceeds 32; banks capped at the same sample bound share one key.

  1. After warmup the cache holds the eager CP2/CP1 tables, and the CP1 graphs read them in place.
  2. The CP8 capture evicts the CP2 tables.
  3. The CP4 capture evicts every CP1 table, so all CP1 graphs now read freed memory, which empty_cache() (4484) may release.

CP1 microbatches then compute indexer q/k with garbage frequencies and get a wrong top-k with no error, or hit an illegal address. Separately, a table built on a capture miss lives in the slot pool, and slots >= 1 read it without TE#3353 protection; MLA has this problem too.

Suggestion: Own the tables in RotaryEmbedding. During is_graph_capturing(), use a per-module dict instead of the lru: warmup fills it and _clear_cuda_graph_state drops it. Raise if a stream capture would build a new table. Add a test with more than 32 keys that compares DSA replay with eager.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. On the reviewed head, the class-wide lru_cache(maxsize=32) was the only owner of the RoPE tables that the DSA indexer (and, with RoPE fusion off, AbsorbedMLA/DSv4/CSA) reads inside the graphs. With more than 32 layer×bank keys, a table that an earlier bank still reads could be evicted, and a capture-time miss allocated the table in the graph pool.

Two owners now keep these tables alive:

  • Upstream Fix potential positional embedding tensor reuse for train/rollout 0-mismatch RL #8358 (in dev) registers every table read during stream capture as a capture constant of the TE graphs.
  • This PR gives every RotaryEmbedding (YaRN included) a per-module table cache while TECudaGraphHelper captures. The warmups of all banks fill it before the first stream capture; capture only reads it and raises on a miss. The cache is released by delete_cuda_graphs() and on capture abort.

Tests:

  • test_attention_variant_dsa.py::TestDSAIndexer::test_graph_replay_keeps_rope_table_beyond_lru_capacity captures DSAIndexer.forward_before_topk (default non-interleaved RoPE). It then pushes the table out of the LRU with 40 other lengths, fills the freed memory with NaN, and checks that replay matches eager. With both owners disabled, the same test fails.
  • test_rope.py::test_graph_rope_tables_survive_more_than_lru_capacity covers RotaryEmbedding and YaRN directly and checks that a capture-time miss raises.

The DSA test uses one bank because the number of banks only changes how many keys compete for the LRU, which the test reproduces directly.

"sequence_packing_scheduler, or use thd_tail_padding_policy='append_dummy_seq'."
)

if self.dynamic_context_parallel and self.cuda_graph_impl != "none":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[IMPORTANT Correctness] This block accepts cp_comm_type="all_gather", but TE's THD all-gather CP path keeps per-step metadata it computes in forward only on ctx. thd_cu_seqlens_q_per_step and thd_cu_seqlens_q_padded_per_step are allocated in forward on both THD branches (TE pytorch/attention/dot_product_attention/context_parallel.py:3526-3581 @ #3353), left out of prepare_for_saving (:3932-3940), stored on ctx (:3975-3976) and read back in backward (:4115, :4124, used at :4240-4241). _graph_memory_slots snapshots only saved tensors, and its docstring states that a tensor backward reads from a plain ctx attribute may be overwritten by another variant's replay (TE pytorch/graph.py:3182-3185 @ #3353). These blocks therefore return to the shared pool after their backward capture, and any slot or bank captured later can reuse them.

Trigger: DCP with context_parallel_size>=2 (with static CP=1, TEDotProductAttention never forwards cp_comm_type; see extensions/transformer_engine.py:2499), cp_comm_type="all_gather", attention graphed, PP=2. Put microbatch m on the CP=4 bank and m+1 on the CP=2 bank, which is captured after it; the replay order is F_m, F_{m+1}, B_m. F_{m+1} can overwrite m's per-step cu_seqlens, so B_m runs the cuDNN backward on wrong segment boundaries. The result is wrong dQ/dK/dV or an illegal memory access. Slot wrap within one bank (B1(s1) replaying between F2(s0) and B2(s0)) triggers it too.

Suggestion: Until TE saves these lists via prepare_for_saving, raise here when attention is graphed, context_parallel_size > 1 and any cp_comm_type entry is "all_gather". Exempt DSA, which requires allgather and does not use TE CP attention.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. AttnFuncWithCPAndKVAllGather kept the THD per-step cu_seqlens_q, cu_seqlens_q_padded and cu_seqlens_kv_padded only on ctx, so the slot arenas could not protect them. Rather than rejecting all_gather in Megatron, NVIDIA/TransformerEngine#3353 now saves them through save_for_backward (NVIDIA/TransformerEngine@a9d4df21). test_saved_quantizer_scales.py::test_fp8_attention_slot_replay_matches_eager_with_overlapping_frames runs a CP1 bank and a CP2 all-gather THD bank on shared slots with different pack boundaries. It covers both of your cases: frames that overlap across banks, and the slot wrap F(s0), F(s1), B(s0), F(s0), B(s1). It matches eager on GB200; these cases need SM100 and cuDNN >= 9.23.

"fp8_attention_flag",
("fp8_dot_product_attention", "fp8_multi_head_attention", "custom_recipe"),
)
def test_dynamic_cp_graph_attention_accepts_fp8_with_native_groups(self, fp8_attention_flag):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[IMPORTANT Correctness] This test locks in that DCP graphs accept fp8_dot_product_attention with the default recipe. With FP8 attention, TE's attention backward reads a tensor created in forward that _graph_memory_slots does not protect:

  • CP=1 bank: FusedAttnFunc copies S_quantizer and clones its scale whenever the S quantizer is a Float8Quantizer (TE pytorch/attention/dot_product_attention/backends.py:1990-1992 @ ci: Concurrency control for merge-queue #3353). The clone lives only in ctx.backward_objects (:2415) and is passed to fused_attn_bwd as the S scale (:2287). The S quantizer is a Float8Quantizer under delayed, and also under tensorwise, where TE builds a separate delayed-scaling recipe for S/dP (recipe table in TE dot_product_attention.py, FP8CS/FP8CS row).
  • CP>=2 banks: the P2P/AG/A2A paths clone it into ctx.S_quantizer under delayed (TE context_parallel.py:2421-2424, :3990-3993, :5049-5052).

Only saved tensors get arena snapshots (TE pytorch/graph.py:3182-3185). Any graph captured after this graph's backward can therefore reuse the clone's pool block. MXFP8 attention has no such clone: its S quantizer is None.

Trigger: sm100+ with cuDNN >= 9.23 (needed for FP8 THD), DCP, PP=2, fp8="hybrid" with fp8_recipe delayed or tensorwise, fp8_dot_product_attention=True, attention graphed, default NVTE_FP8_DPA_BWD=1. The replay order is F2(s0), B1(s1), F3(s1), B2(s0). B(s1) was captured after B(s0) and can overwrite F2's clone. B2 then quantizes P with a wrong scale and returns bad dQ/dK/dV without any error. Microbatches on different banks interleave the same way.

Suggestion: In transformer_config.py:5676-5692, reject fp8_dot_product_attention/fp8_multi_head_attention when attention is graphed, unless the attention recipe is MXFP8 or NVTE_FP8_DPA_BWD=0. Flip the non-MXFP8 cases of this test and add an accepted MXFP8 case. The cleaner long-term fix is for TE to save the cloned scale through save_for_backward; the check can then be removed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The forward scale snapshots lived only on ctx: S in FusedAttnFunc, and QKV/O/S in the CP P2P, all-gather and A2A paths. Instead of rejecting FP8 DPA in Megatron, NVIDIA/TransformerEngine#3353 saves them through save_for_backward (NVIDIA/TransformerEngine@a9d4df21). For your tensorwise case, the CP paths now snapshot every Float8Quantizer scale, not only under a delayed global recipe. Two tests match eager:

  • TE test_fp8_attention_slot_replay_matches_eager_with_overlapping_frames, with a different scale per frame, covering delayed and current scaling, SBHD and THD, no CP, P2P, A2A and all-gather, with offload off and on.
  • Megatron test_dcp_partial_graph_offload_pipeline_matches_eager, with FP8 DPA over PP2 and PP2/VPP2.

Comment on lines +5676 to +5692
if self.dynamic_context_parallel and self.cuda_graph_impl != "none":
if self.cuda_graph_impl != "transformer_engine":
raise ValueError("Dynamic CP supports only layer-wise TE CUDA graphs.")
if not self.cuda_graph_dynamic_microbatches:
raise ValueError("Dynamic CP CUDA graphs require dynamic microbatch slots.")
if self.delay_wgrad_compute or self.overlap_moe_expert_parallel_comm:
raise ValueError("Dynamic CP graphs do not support delayed wgrad or EP overlap.")
if self.moe_paged_stash:
raise ValueError(
"Dynamic CP graphs do not support moe_paged_stash: its capture bookkeeping "
"does not cover the per-CP-size graph banks."
)
if self.cuda_graph_modules and CudaGraphModule.mamba in self.cuda_graph_modules:
raise ValueError(
"Dynamic CP graphs capture TransformerLayer scopes only; remove 'mamba' from "
"cuda_graph_modules."
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[IMPORTANT Validation] This block does not reject fine-grained activation offloading with offload modules inside the captured scope. That combination cannot capture under TE#3353 slot mode.

post_warmup_hook (cuda_graphs.py:4016-4018) re-enables offload after the last slot template's warmup. During capture, PipelineOffloadManager.__enter__ (fine_grained_activation_offload.py:849) then pushes its own saved_tensors_hooks inside TE's capture_saved_tensors hook (TE transformer_engine/pytorch/graph.py:2254 @ #3353). PyTorch applies only the innermost pair, so saves inside qkv_linear/core_attn/attn_proj go to tensor_push and never reach pack_saved_tensor. The warmup plan was recorded with offload disabled, so it counted those saves.

Trigger: DCP + TE graphs with cuda_graph_modules=[attn] and offload_modules=[core_attn, attn_proj], the documented attention-offload setup. The first captured forward fails with "changed forward saved-tensor layout during capture" or "saved N forward tensors during capture, but saved M during warmup" (TE graph.py:2213/2257). Neither message mentions offloading.

Suggestion: Raise here when fine_grained_activation_offloading has an offload module inside the graph scope, i.e. when any layer's offload_scope_in_cuda_graph() is true (attention offload modules with attn, mlp_norm with mlp; transformer_layer.py:2486-2526). That predicate returns False for whole-layer capture (empty cuda_graph_modules), which today only warns; under DCP, reject that combination here as well. Name the slot-arena conflict in the error and add a config test. Supporting the combination later would need the offload pack hook to delegate to TE's slot hook.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Capture ran PipelineOffloadManager's hooks inside TE's capture_saved_tensors, so offloaded saves bypassed the slot plan. In 6bbffab, graph-scoped offload goes through TE's slot hook (NVIDIA/TransformerEngine#3353) instead. Warmup and capture now use one hook pair with a per-scope checker, and TE copies the selected saves into pinned host arenas and reloads them in backward. Whole-layer capture with offloading is now rejected for dynamic-CP graphs, as you suggested (6b43bf5). test_dcp_partial_graph_offload_pipeline_matches_eager runs your setup (cuda_graph_modules=[attn], offload_modules=[core_attn, attn_proj]) over PP2 and PP2/VPP2, with CP1/CP2 banks and slot wrap, and matches eager offloading. Limitation: the policy is frozen per module scope, so graph offload can save less than eager. Memory and throughput are not benchmarked yet.

Comment on lines +4635 to +4643
sample_args, kwargs = self._get_cuda_graph_input_data()
if expected_num_microbatches is None:
expected_num_microbatches = self.num_microbatches
elif self.num_microbatches != expected_num_microbatches:
raise RuntimeError(
"Dynamic-CP capture changed slot count across variants: "
f"{expected_num_microbatches} -> {self.num_microbatches}."
)
capture_banks.append((cp_size, cp_group, sample_args, kwargs))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[IMPORTANT Memory] Every pass of this loop calls _get_cuda_graph_input_data(). Its _get_sample_arguments allocates fresh leading hidden_states samples ([mslpr/TP, 1, hidden], or n*hidden for mHC) for every peak-live layer input of the capture order. All B per-bank lists stay referenced from capture_banks until make_graphed_callables returns: they are written back at 4407 and released at 4653.

TE#3353 stages leading inputs per (saved color, layout) (TE transformer_engine/pytorch/graph.py:1594-1605 @ #3353). The colors are shared by all banks, so only about one storage per color survives. The remaining ~(B-1)/B of the samples are dead weight through TE warmup and capture, which is exactly when TE allocates its arenas. The leading-input shape does not depend on CP size, so separate tensors per bank are not needed.

Trigger: DPxCP=64 (7 banks), PP=8/VPP=4, 2 layers per chunk, hidden=8192, TP=8+SP, mslpr=8192. Each sample is 16 MiB and PP rank 0 needs 78 per bank. That holds ~8.5 GiB against the ~1.2 GiB that survives: about 7 GiB of capture-only peak, and 4x that at TP=2. Capture can OOM in a run whose steady state fits.

Suggestion: Allocate the leading samples once, for the first bank. Reuse the same tensor for the matching (chunk, slot, layer) in every other bank, and regenerate only the CP-dependent keyword inputs. Alternatively, drop the non-canonical leading inputs before calling TE.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, fixed in 6bbffab with your second option. Each later bank reuses the first bank's leading hidden_states samples through TECudaGraphHelper._reuse_dynamic_cp_leading_samples, which checks that layouts and aliases match. CP-dependent args stay per bank, so only one sample set is live during warmup and capture. test_thd_cuda_graph.py::TestDynamicMicrobatchSlots::test_dynamic_cp_capture_reuses_leading_samples_across_banks covers the reuse, and the multi-bank replay tests still match eager. Capture peak memory at your scale is not measured yet.

Comment thread megatron/core/datasets/data_schedule.py Outdated
if scheduler_type == 'dp_balanced'
else getattr(config, 'thd_max_packed_sequences', None)
)
scheduler_max_num_seqs = _get_scheduler_max_real_num_seqs(config)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SUGGESTION Documentation] With this change every packing scheduler, not only dp_balanced, gets thd_max_packed_sequences - 1 when a dummy tail is appended. DefaultDynamicCPScheduler now enforces that cap per CP subgroup (line 467; data_schedule_utils.py:873-877, :1013-1015); on the base, DCP ignored it.

The field docstring, which is also the --thd-max-packed-sequences help text (transformer_config.py:1531-1540), was not updated:

  • It still says "The dp_balanced packing scheduler reserves that dummy slot".
  • Its sizing note ("setting it too large just allocates a slightly larger cu_seqlens buffer") no longer covers dynamic CP. With TE graphs, train() reuses the value as thd_max_subsamples_per_item for non-varlen data (training.py:7895-7897). With PP>1, that sets the packed-microbatch range _validate_cuda_graph_microbatch_count enforces at runtime.

A DCP user tuning this for eager packing now changes both the pack cap and, with graphs, the accepted microbatch range.

Suggestion: Update the docstring: both schedulers cap real sequences per pack at this value (minus one with dummy-tail padding), applied per CP subgroup under dynamic CP. While train() still derives the per-item bound from this field, also say that with dynamic-CP CUDA graphs it bounds the accepted packed microbatch count.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, updated in 6bbffab. The docstring and the --thd-max-packed-sequences help now say that both dp_balanced and default_dynamic_cp cap real sequences at this value (minus one with a dummy tail), and that dynamic CP applies the cap per CP subgroup. The field no longer sets the captured graph range (see the source-subsample thread).

xiaoyao0115 and others added 7 commits November 17, 2026 21:25
Dynamic context parallelism (DCP) picks a CP size for every microbatch, and CP communicators
are capture-time constants. TECudaGraphHelper therefore captures one graph bank per
power-of-two CP size, from --min-dynamic-context-parallel-size to DP x CP, each with the real
process group of that size. All banks share one TE memory pool through
make_graphed_callables(_graph_memory_slots=...) (NVIDIA/TransformerEngine#3353), so graph
memory does not grow with the number of banks.

- Slot plan: the physical slot ring is sized from the liveness union of every PP/VPP order the
  DCP scheduler can produce. Saved-tensor and returned-gradient arenas are colored so that
  frames that are live at the same time never share storage.
- Replay: a layer selects its bank (graphs, process group, static inputs) from
  PackedSeqParams.local_cp_size, including the mHC attention split and the hybrid wrapper,
  and checks that the microbatch's runtime CP group is the one the bank captured. A CP-off
  microbatch may carry its singleton group in cp_group or in cp_singleton_group.
  Router-only captures get a DCP PackedSeqParams so they use the bank's group.
- Capture state: THD capture seeds a bounded RoPE table and cu_seqlens per bank, and eager RoPE
  uses the same bound under DCP. During training the packed microbatch count is checked
  against the captured range, and the DCP scheduler honors thd_max_packed_sequences, so a
  pack never exceeds the captured cu_seqlens capacity. Capture can be retried, and
  capture-only state is released on failure and on delete.
- Config rules: DCP with CUDA graphs requires TE graphs and power-of-two DP x CP, and rejects
  delayed wgrad, EP overlap, moe_paged_stash and mamba scopes.

Runs without DCP keep dev behavior apart from fixes on the paths they share:
- The metadata-only THD padding path (PP stages without tokens) derives its padding target
  from the CP-local length, as the tensor path already does.
- With CP=1 the dp_balanced scheduler gathers on the DPxCP communicator, which has the same
  ranks in the same order as DP.
- Packed THD capture with fixed CP>1 seeds the full RoPE table and graph cu_seqlens instead of
  the CP-sliced table.
- After an attention-only graph replay, the eager MLP receives the real PackedSeqParams.
- PP stages without graphable layers still join the all-reduce that sizes the dynamic
  microbatch slot ring, so stages with graphs do not wait for them.

Co-authored-by: HaochenYuan <haocheny@nvidia.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Graph slot memory protects only tensors saved for backward. With dynamic-CP graph banks, a
replay of another bank can overwrite tensors that a forward kept in ctx attributes before the
matching backward runs. The CSA sparse attention functions, the cuDNN sparse attention
functions, FusedDSAIndexerLoss and the fused MLA RoPE in-place, concat and KV-split functions
now pass that metadata through save_for_backward.

Dynamic-CP capture walks several CP banks in sequence. It relies on CSA capture reusing any
warmed-up compact-indexer workspace that matches the geometry (NVIDIA#8104); the tests cover this
for BSHD and THD.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Review follow-ups for the dynamic-CP graph banks, and fine-grained activation offloading
and FP8 attention inside captured scopes.

- RoPE tables: a table read inside a graph must outlive the graph, but the class-wide LRU
  cache can evict it. While TE captures the banks, every RotaryEmbedding keeps the tables
  its warmups compute in a per-module cache; stream capture only reads that cache (a miss
  raises), and the tables are released when the graphs are deleted or capture aborts.
  Local graphs keep dev's capture-constant ownership. A DSA indexer graph test replays
  correctly after more than 32 other tables evict its table from the shared cache.
- Capture memory: banks after the first reuse the first bank's leading hidden-state samples
  when their layouts and alias topology match, instead of keeping one set per bank until
  make_graphed_callables returns.
- Microbatch bound: the capture range for real SFT items is derived from seq_length and the
  dataset's CP/SP padding divisor, independently of thd_max_packed_sequences, which only
  caps output packs.
- Offloading: offload scopes inside a captured slot graph hand their saves to TE's slot
  hooks (_graph_memory_offload_context) instead of pushing the offload manager's own
  saved-tensor hooks inside TE's. The policy is frozen per module scope from eager warmup;
  attention, MLA, DSv4, experts and hybrid layers pass their module as the scope owner.
- FP8 attention: TE#3353 now saves quantizer scale snapshots through autograd, so the
  dynamic-CP graph tests also cover fp8_dot_product_attention.
- Docs, typing and the thd_max_packed_sequences description follow the new behavior.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
The style guide asks for double quotes in new code. Convert the single-quoted literals
on lines this PR adds; lines moved unchanged from dev keep their quoting. No behavior
change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Dynamic-CP graphs bound the RoPE table to the captured length while the packed max_seqlen
can include a padding tail. apply_rotary_pos_emb then skips TE's fused THD RoPE, which
needs the table to cover max_seqlen, and uses the unfused THD path. Replay that fallback
forward and backward byte for byte for CP1 and both CP2 ranks, with a sequence that runs
past the table, and note it in the rope_dispatch manifest entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
…undary

- With pipeline parallelism, dynamic-CP graphs size their schedule from a bound on the
  sequences in one dataset item. GPT items with --dataloader-inter-document-masking can
  split into up to seq_length documents, so argument validation now rejects that
  combination instead of failing a valid step later. SFT keeps its derived bound, and
  varlen and FIM items hold one sequence. The SFT bound's comment now explains the +1.
- Fine-grained activation offloading with whole-layer TE capture does not install the
  per-module offload boundary, which only had a warning. Dynamic-CP graphs now reject it
  and require partial cuda_graph_modules scopes.
- The context-parallel and offloading guides state both limits, and the offloading guide
  describes the snapshot copies of asynchronous D2H.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Capture built a PP/VPP order for every microbatch count up to the scheduler bound and colored
all of them, so its cost grew quadratically with the bound. The SFT source-item bound can make
that tens of thousands of microbatches, which would stall capture.

Longer schedules only repeat the live sets of shorter ones: a reachable count's schedule table
is a prefix of a longer one's, and past the warmup each forward/backward pair recurs on the same
model chunks and physical slots every lcm(slots, VPP group size) microbatches. The proof now
enumerates counts up to the warmup plus that period: first to size the slot ring, then, after
the PP all-reduce, to color the ring's frames. The runtime contract still covers every count up
to the scheduler bound.

The slot-color test proves over the truncated counts, as capture does, and replays every
reachable count up to twice the largest limit with the independent overlap check, now also with
a VPP group size of twice the PP size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Tailai Ma <tailaim@nvidia.com>
@xiaoyao0115

Copy link
Copy Markdown
Contributor Author

Re the [P2] in the review body (short-table THD RoPE determinism test):

Thanks, added in 2c0fd65: test_te_wrappers.py::test_thd_short_rope_table_replays_fwd_bwd, adapted from your suggestion. It covers CP1 and both CP2 ranks, with fused THD RoPE stubbed to raise, a 1024-row table with max_seqlen=4096, and a rotary dim smaller than the head dim. Output and input gradient are byte-identical across three replays on H100 and GB200. The full-table and raw mRoPE tests are unchanged.

We keep determinism-exempt for the five files the gate still flags, none of which changes arithmetic. fused_mla_yarn_rope_apply.py, dsa.py and csa_utils/fused_sparse_attention.py move per-pack backward metadata from ctx into save_for_backward. attention.py and moe/experts.py only pass owner=self to the offload interface.

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants