Repository navigation
[dev] Share CUDA graph slots across dynamic CP variants - #6479
xiaoyao0115 wants to merge 7 commits into
Conversation
51b9fad to
33082d6
Compare
21f2169 to
3de21ef
Compare
|
/ok to test 2a1d969 |
|
/ok to test a24ea4d |
|
/ok to test 7e6ba4c |
7e6ba4c to
8bb6345
Compare
|
/ok to test 8bb6345 |
69f978a to
cdfa7c2
Compare
|
/ok to test cdfa7c2 |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
Completed the light review across all changed files and their directly affected paths. One scheduling-bound correctness issue, one determinism-coverage gap, and two unenforced style-guide violations remain. Tests were inspected, not executed; Transformer Engine’s external slot allocator was not available for inspection.
[P2] Add bit-exact coverage for the new THD RoPE dispatch branch
Failure — coverage: The new short-frequency-table check changes dispatch from TE to the unfused implementation. The added fusion tests check forward closeness and graph replay, but not bit-exact forward/backward repeatability. No determinism test changes accompany this dispatch change, despite the required kernel-testing contract. The dispatch is already registered against the TE wrapper tests.
Correction: Add a short-table forward/backward replay case to the registered test module. Suggested addition, not executed; verify before adopting:
@pytest.mark.parametrize("cp_size, cp_rank", [(1, 0), (2, 0), (2, 1)])
def test_thd_short_rope_replays_fwd_bwd(monkeypatch, cp_size, cp_rank):
from megatron.core.models.common.embeddings import rope_utils
seeded()
config = _config(apply_rope_fusion=True)
group = SimpleNamespace(size=lambda: cp_size, rank=lambda: cp_rank)
cu = torch.tensor([0, 4, 8], dtype=torch.int32, device="cuda")
freqs = torch.randn(4, 1, 1, 16, device="cuda")
t = torch.randn(
8 // cp_size, 3, 20, device="cuda",
dtype=torch.bfloat16, requires_grad=True,
)
def reject_te(*args, **kwargs):
raise AssertionError("short THD tables must bypass TE")
monkeypatch.setattr(rope_utils, "fused_apply_rotary_pos_emb_thd", reject_te)
def apply(t):
return rope_utils.apply_rotary_pos_emb(
t, freqs, config, cu_seqlens=cu, cp_group=group, max_seqlen=8
)
assert_replays_bit_exact(
apply, (t,), replays=3, backward=True, contention=True,
what=f"short THD RoPE CP{cp_size}/rank{cp_rank}",
)Preservation: Retain the existing full-table SBHD/THD fused-RoPE replay cases. The short materialized-table fallback must not redirect raw three-axis mRoPE or change their existing fusion/parity coverage.
Acceptance — proposed: For all three CP cases above, TE must not be called, and every output and input gradient must be byte-identical across replays. Existing full-table fused cases and the new forward/graph-parity tests must continue passing.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| # datasets yield pre-packed items of up to thd_max_packed_sequences sequences. | ||
| max_subsamples_per_item = 1 | ||
| if not args.use_varlen_dataset and args.thd_max_packed_sequences is not None: | ||
| max_subsamples_per_item = args.thd_max_packed_sequences |
There was a problem hiding this comment.
[P2] Bound source subsamples independently of the destination pack capacity
Failure: thd_max_packed_sequences limits each repacked output, not the number of sequences in a source dataset item. SFTDataset packs conversations up to sequence_length without consulting this setting, and the scheduler unpacks every nonempty interval.
A concrete supported case is TP1, PP2, static CP1, DP2, GBS2, seq_length=256, local token capacity 64, and thd_max_packed_sequences=2 with dummy-tail padding. Each of the two source items can contain intervals [80, 80, 96]. Every interval needs CP2; the one-real-sequence output cap therefore produces six microbatches. This assignment instead declares two subsamples per item, so the capture bound is four. After capture, runtime validation rejects this valid step instead of training it.
Correction: Change _get_dynamic_cp_cuda_graph_helper_kwargs to pass an independently justified source-item subsample bound. For in-tree SFT, derive it from source pack length and the dataset’s CP/SP padding granularity, or expose the dataset’s actual maximum through the helper plumbing. Do not use the post-repacking capacity as that bound, and retain the runtime liveness guard. The tight implementation depends on how the source-dataset bound is exposed, so substituting another unexplained constant would not be a complete fix.
Preservation: Varlen items must still contribute exactly one sequence to this calculation. Fixed-CP/non-THD helper construction must still work with thd_max_packed_sequences=None. Destination packs must retain their one-real-sequence-plus-dummy limit in this example.
Acceptance — proposed, not executed: Feed six lengths [80,80,96,80,80,96] to DefaultDynamicCPScheduler with two total DCP ranks, local capacity 64 and max_num_seqs=1; assert six output microbatches. Build the capture contract from the corresponding two-item source setup and assert its bound covers six and runtime validation accepts six. Retain rejection above the proven bound and the parsed-default/varlen assertions in the plumbing tests, updating the non-varlen expectation to the independently derived source bound.
There was a problem hiding this comment.
Agreed, fixed in 6bbffab. The capture bound no longer uses thd_max_packed_sequences. For --sft data it is seq_length // d + 1 per source item, where d is the CP/SP padding divisor; other data keeps 1. Pre-packed data with --dataloader-inter-document-masking has no usable bound, so argument validation now rejects it when dynamic-CP graphs run with PP>1 (6b43bf5). test_sft_source_subsample_bound_covers_dynamic_cp_repacking runs your example and gets six microbatches, all covered by the bound.
| class _DynamicCPCaptureCallable(torch.nn.Module): | ||
| """Set one dynamic-CP context immediately before capturing a wrapped layer.""" | ||
|
|
||
| def __init__(self, module, context_setter, context): |
There was a problem hiding this comment.
[P3] Annotate the new capture-wrapper constructor
Failure — static: The new constructor lacks the -> None annotation required by the protected style guide. This is a source-style violation, not a runtime failure.
Correction: Replace only the signature:
| def __init__(self, module, context_setter, context): | |
| def __init__(self, module, context_setter, context) -> None: |
Preservation: Leave module registration, grouped-prefix exposure, capture context and training state unchanged.
Acceptance — static/proposed: Confirm the constructor is annotated and its body is unchanged. Retain the grouped-prefix test’s assertions that model state-dict keys do not change and the temporary callable exposes the prefix’s parameters and buffer. The suggestion was not executed; verify it before applying.
There was a problem hiding this comment.
Done in 6bbffab: _DynamicCPCaptureCallable.__init__ is annotated -> None.
|
|
||
| def _validate_cuda_graph_microbatch_count(config, num_microbatches: int) -> None: | ||
| """Require runtime packing to stay within the captured schedule proof.""" | ||
| limit = getattr(config, '_cuda_graph_num_microbatches', None) |
There was a problem hiding this comment.
[P3] Use double quotes for newly introduced string literals
Failure — static: The new helper uses single-quoted attribute names, contrary to the style guide’s double-quote rule. The same issue occurs in newly added capture-state literals in cuda_graphs and the RoPE-bound helper. Black explicitly preserves quote style, so formatting does not repair this convention.
Correction: Normalize newly added literals to double quotes without restyling untouched code. For this line:
| limit = getattr(config, '_cuda_graph_num_microbatches', None) | |
| limit = getattr(config, "_cuda_graph_num_microbatches", None) |
Apply the same delimiter-only correction to the other newly introduced literals.
Preservation: Keep string values, attribute names, dictionary keys and error messages identical; this must not alter capture-state lookup or validation behavior.
Acceptance — static: Inspect the added lines for double-quoted literals and verify their decoded values are unchanged. No runtime behavior change or new runtime test is required. Verify the suggestion before applying.
|
/ok to test 52953e2 |
cdfa7c2 to
52953e2
Compare
| def _retain_cuda_graph_rope_tensors(self, *rope_tensors: Optional[torch.Tensor]) -> None: | ||
| """Retain MLA-internal RoPE allocations used by TE CUDA graphs.""" | ||
| if getattr(self.config, 'cuda_graph_impl', 'none') != 'transformer_engine': | ||
| return | ||
|
|
||
| from megatron.core.transformer.cuda_graphs import is_graph_capturing | ||
|
|
||
| if not is_graph_capturing(): | ||
| return | ||
| refs = self.__dict__.setdefault('_cuda_graph_static_tensor_refs', {}) | ||
| refs.update((id(tensor), tensor) for tensor in rope_tensors if tensor is not None) |
There was a problem hiding this comment.
[CRITICAL Memory safety] This pins RoPE tables only for MLASelfAttention. Other in-graph producers read the table that RotaryEmbedding/YarnRotaryEmbedding.forward return, and its only owner is a class-wide @lru_cache(maxsize=32) (rotary_pos_embedding.py:181, yarn_rotary_pos_embedding.py:159). These producers are:
DSAIndexer.forward_before_topkwith the defaultdsa_indexer_rope_interleaved=False(dsa.py:2148-2151)- with fusion off: AbsorbedMLA (absorbed_mla.py:546-548), DSv4 and CSA
Under DCP every bank adds one key per layer (min(max_seqlen_per_dp_cp_rank*cp, sample bound)). TE warms the largest CP first (TE transformer_engine/pytorch/graph.py:935-939 @ #3353), but the smallest CP is captured first (cuda_graphs.py:4112, 4350).
Trigger: DSA, 16 layers per stage, DPxCP=8, and seq_length >= 8 * max_seqlen_per_dp_cp_rank, so every bank keeps its own table length and there are 64 keys. In general it fires once layers per stage × distinct per-bank table lengths exceeds 32; banks capped at the same sample bound share one key.
- After warmup the cache holds the eager CP2/CP1 tables, and the CP1 graphs read them in place.
- The CP8 capture evicts the CP2 tables.
- The CP4 capture evicts every CP1 table, so all CP1 graphs now read freed memory, which
empty_cache()(4484) may release.
CP1 microbatches then compute indexer q/k with garbage frequencies and get a wrong top-k with no error, or hit an illegal address. Separately, a table built on a capture miss lives in the slot pool, and slots >= 1 read it without TE#3353 protection; MLA has this problem too.
Suggestion: Own the tables in RotaryEmbedding. During is_graph_capturing(), use a per-module dict instead of the lru: warmup fills it and _clear_cuda_graph_state drops it. Raise if a stream capture would build a new table. Add a test with more than 32 keys that compares DSA replay with eager.
There was a problem hiding this comment.
Agreed. On the reviewed head, the class-wide lru_cache(maxsize=32) was the only owner of the RoPE tables that the DSA indexer (and, with RoPE fusion off, AbsorbedMLA/DSv4/CSA) reads inside the graphs. With more than 32 layer×bank keys, a table that an earlier bank still reads could be evicted, and a capture-time miss allocated the table in the graph pool.
Two owners now keep these tables alive:
- Upstream Fix potential positional embedding tensor reuse for train/rollout 0-mismatch RL #8358 (in dev) registers every table read during stream capture as a capture constant of the TE graphs.
- This PR gives every
RotaryEmbedding(YaRN included) a per-module table cache whileTECudaGraphHelpercaptures. The warmups of all banks fill it before the first stream capture; capture only reads it and raises on a miss. The cache is released bydelete_cuda_graphs()and on capture abort.
Tests:
test_attention_variant_dsa.py::TestDSAIndexer::test_graph_replay_keeps_rope_table_beyond_lru_capacitycapturesDSAIndexer.forward_before_topk(default non-interleaved RoPE). It then pushes the table out of the LRU with 40 other lengths, fills the freed memory with NaN, and checks that replay matches eager. With both owners disabled, the same test fails.test_rope.py::test_graph_rope_tables_survive_more_than_lru_capacitycoversRotaryEmbeddingand YaRN directly and checks that a capture-time miss raises.
The DSA test uses one bank because the number of banks only changes how many keys compete for the LRU, which the test reproduces directly.
| "sequence_packing_scheduler, or use thd_tail_padding_policy='append_dummy_seq'." | ||
| ) | ||
|
|
||
| if self.dynamic_context_parallel and self.cuda_graph_impl != "none": |
There was a problem hiding this comment.
[IMPORTANT Correctness] This block accepts cp_comm_type="all_gather", but TE's THD all-gather CP path keeps per-step metadata it computes in forward only on ctx. thd_cu_seqlens_q_per_step and thd_cu_seqlens_q_padded_per_step are allocated in forward on both THD branches (TE pytorch/attention/dot_product_attention/context_parallel.py:3526-3581 @ #3353), left out of prepare_for_saving (:3932-3940), stored on ctx (:3975-3976) and read back in backward (:4115, :4124, used at :4240-4241). _graph_memory_slots snapshots only saved tensors, and its docstring states that a tensor backward reads from a plain ctx attribute may be overwritten by another variant's replay (TE pytorch/graph.py:3182-3185 @ #3353). These blocks therefore return to the shared pool after their backward capture, and any slot or bank captured later can reuse them.
Trigger: DCP with context_parallel_size>=2 (with static CP=1, TEDotProductAttention never forwards cp_comm_type; see extensions/transformer_engine.py:2499), cp_comm_type="all_gather", attention graphed, PP=2. Put microbatch m on the CP=4 bank and m+1 on the CP=2 bank, which is captured after it; the replay order is F_m, F_{m+1}, B_m. F_{m+1} can overwrite m's per-step cu_seqlens, so B_m runs the cuDNN backward on wrong segment boundaries. The result is wrong dQ/dK/dV or an illegal memory access. Slot wrap within one bank (B1(s1) replaying between F2(s0) and B2(s0)) triggers it too.
Suggestion: Until TE saves these lists via prepare_for_saving, raise here when attention is graphed, context_parallel_size > 1 and any cp_comm_type entry is "all_gather". Exempt DSA, which requires allgather and does not use TE CP attention.
There was a problem hiding this comment.
Agreed. AttnFuncWithCPAndKVAllGather kept the THD per-step cu_seqlens_q, cu_seqlens_q_padded and cu_seqlens_kv_padded only on ctx, so the slot arenas could not protect them. Rather than rejecting all_gather in Megatron, NVIDIA/TransformerEngine#3353 now saves them through save_for_backward (NVIDIA/TransformerEngine@a9d4df21). test_saved_quantizer_scales.py::test_fp8_attention_slot_replay_matches_eager_with_overlapping_frames runs a CP1 bank and a CP2 all-gather THD bank on shared slots with different pack boundaries. It covers both of your cases: frames that overlap across banks, and the slot wrap F(s0), F(s1), B(s0), F(s0), B(s1). It matches eager on GB200; these cases need SM100 and cuDNN >= 9.23.
| "fp8_attention_flag", | ||
| ("fp8_dot_product_attention", "fp8_multi_head_attention", "custom_recipe"), | ||
| ) | ||
| def test_dynamic_cp_graph_attention_accepts_fp8_with_native_groups(self, fp8_attention_flag): |
There was a problem hiding this comment.
[IMPORTANT Correctness] This test locks in that DCP graphs accept fp8_dot_product_attention with the default recipe. With FP8 attention, TE's attention backward reads a tensor created in forward that _graph_memory_slots does not protect:
- CP=1 bank:
FusedAttnFunccopiesS_quantizerand clones itsscalewhenever the S quantizer is aFloat8Quantizer(TEpytorch/attention/dot_product_attention/backends.py:1990-1992@ ci: Concurrency control for merge-queue #3353). The clone lives only inctx.backward_objects(:2415) and is passed tofused_attn_bwdas the S scale (:2287). The S quantizer is aFloat8Quantizerunderdelayed, and also undertensorwise, where TE builds a separate delayed-scaling recipe for S/dP (recipe table in TEdot_product_attention.py, FP8CS/FP8CS row). - CP>=2 banks: the P2P/AG/A2A paths clone it into
ctx.S_quantizerunderdelayed(TEcontext_parallel.py:2421-2424,:3990-3993,:5049-5052).
Only saved tensors get arena snapshots (TE pytorch/graph.py:3182-3185). Any graph captured after this graph's backward can therefore reuse the clone's pool block. MXFP8 attention has no such clone: its S quantizer is None.
Trigger: sm100+ with cuDNN >= 9.23 (needed for FP8 THD), DCP, PP=2, fp8="hybrid" with fp8_recipe delayed or tensorwise, fp8_dot_product_attention=True, attention graphed, default NVTE_FP8_DPA_BWD=1. The replay order is F2(s0), B1(s1), F3(s1), B2(s0). B(s1) was captured after B(s0) and can overwrite F2's clone. B2 then quantizes P with a wrong scale and returns bad dQ/dK/dV without any error. Microbatches on different banks interleave the same way.
Suggestion: In transformer_config.py:5676-5692, reject fp8_dot_product_attention/fp8_multi_head_attention when attention is graphed, unless the attention recipe is MXFP8 or NVTE_FP8_DPA_BWD=0. Flip the non-MXFP8 cases of this test and add an accepted MXFP8 case. The cleaner long-term fix is for TE to save the cloned scale through save_for_backward; the check can then be removed.
There was a problem hiding this comment.
Agreed. The forward scale snapshots lived only on ctx: S in FusedAttnFunc, and QKV/O/S in the CP P2P, all-gather and A2A paths. Instead of rejecting FP8 DPA in Megatron, NVIDIA/TransformerEngine#3353 saves them through save_for_backward (NVIDIA/TransformerEngine@a9d4df21). For your tensorwise case, the CP paths now snapshot every Float8Quantizer scale, not only under a delayed global recipe. Two tests match eager:
- TE
test_fp8_attention_slot_replay_matches_eager_with_overlapping_frames, with a different scale per frame, covering delayed and current scaling, SBHD and THD, no CP, P2P, A2A and all-gather, with offload off and on. - Megatron
test_dcp_partial_graph_offload_pipeline_matches_eager, with FP8 DPA over PP2 and PP2/VPP2.
| if self.dynamic_context_parallel and self.cuda_graph_impl != "none": | ||
| if self.cuda_graph_impl != "transformer_engine": | ||
| raise ValueError("Dynamic CP supports only layer-wise TE CUDA graphs.") | ||
| if not self.cuda_graph_dynamic_microbatches: | ||
| raise ValueError("Dynamic CP CUDA graphs require dynamic microbatch slots.") | ||
| if self.delay_wgrad_compute or self.overlap_moe_expert_parallel_comm: | ||
| raise ValueError("Dynamic CP graphs do not support delayed wgrad or EP overlap.") | ||
| if self.moe_paged_stash: | ||
| raise ValueError( | ||
| "Dynamic CP graphs do not support moe_paged_stash: its capture bookkeeping " | ||
| "does not cover the per-CP-size graph banks." | ||
| ) | ||
| if self.cuda_graph_modules and CudaGraphModule.mamba in self.cuda_graph_modules: | ||
| raise ValueError( | ||
| "Dynamic CP graphs capture TransformerLayer scopes only; remove 'mamba' from " | ||
| "cuda_graph_modules." | ||
| ) |
There was a problem hiding this comment.
[IMPORTANT Validation] This block does not reject fine-grained activation offloading with offload modules inside the captured scope. That combination cannot capture under TE#3353 slot mode.
post_warmup_hook (cuda_graphs.py:4016-4018) re-enables offload after the last slot template's warmup. During capture, PipelineOffloadManager.__enter__ (fine_grained_activation_offload.py:849) then pushes its own saved_tensors_hooks inside TE's capture_saved_tensors hook (TE transformer_engine/pytorch/graph.py:2254 @ #3353). PyTorch applies only the innermost pair, so saves inside qkv_linear/core_attn/attn_proj go to tensor_push and never reach pack_saved_tensor. The warmup plan was recorded with offload disabled, so it counted those saves.
Trigger: DCP + TE graphs with cuda_graph_modules=[attn] and offload_modules=[core_attn, attn_proj], the documented attention-offload setup. The first captured forward fails with "changed forward saved-tensor layout during capture" or "saved N forward tensors during capture, but saved M during warmup" (TE graph.py:2213/2257). Neither message mentions offloading.
Suggestion: Raise here when fine_grained_activation_offloading has an offload module inside the graph scope, i.e. when any layer's offload_scope_in_cuda_graph() is true (attention offload modules with attn, mlp_norm with mlp; transformer_layer.py:2486-2526). That predicate returns False for whole-layer capture (empty cuda_graph_modules), which today only warns; under DCP, reject that combination here as well. Name the slot-arena conflict in the error and add a config test. Supporting the combination later would need the offload pack hook to delegate to TE's slot hook.
There was a problem hiding this comment.
Agreed. Capture ran PipelineOffloadManager's hooks inside TE's capture_saved_tensors, so offloaded saves bypassed the slot plan. In 6bbffab, graph-scoped offload goes through TE's slot hook (NVIDIA/TransformerEngine#3353) instead. Warmup and capture now use one hook pair with a per-scope checker, and TE copies the selected saves into pinned host arenas and reloads them in backward. Whole-layer capture with offloading is now rejected for dynamic-CP graphs, as you suggested (6b43bf5). test_dcp_partial_graph_offload_pipeline_matches_eager runs your setup (cuda_graph_modules=[attn], offload_modules=[core_attn, attn_proj]) over PP2 and PP2/VPP2, with CP1/CP2 banks and slot wrap, and matches eager offloading. Limitation: the policy is frozen per module scope, so graph offload can save less than eager. Memory and throughput are not benchmarked yet.
| sample_args, kwargs = self._get_cuda_graph_input_data() | ||
| if expected_num_microbatches is None: | ||
| expected_num_microbatches = self.num_microbatches | ||
| elif self.num_microbatches != expected_num_microbatches: | ||
| raise RuntimeError( | ||
| "Dynamic-CP capture changed slot count across variants: " | ||
| f"{expected_num_microbatches} -> {self.num_microbatches}." | ||
| ) | ||
| capture_banks.append((cp_size, cp_group, sample_args, kwargs)) |
There was a problem hiding this comment.
[IMPORTANT Memory] Every pass of this loop calls _get_cuda_graph_input_data(). Its _get_sample_arguments allocates fresh leading hidden_states samples ([mslpr/TP, 1, hidden], or n*hidden for mHC) for every peak-live layer input of the capture order. All B per-bank lists stay referenced from capture_banks until make_graphed_callables returns: they are written back at 4407 and released at 4653.
TE#3353 stages leading inputs per (saved color, layout) (TE transformer_engine/pytorch/graph.py:1594-1605 @ #3353). The colors are shared by all banks, so only about one storage per color survives. The remaining ~(B-1)/B of the samples are dead weight through TE warmup and capture, which is exactly when TE allocates its arenas. The leading-input shape does not depend on CP size, so separate tensors per bank are not needed.
Trigger: DPxCP=64 (7 banks), PP=8/VPP=4, 2 layers per chunk, hidden=8192, TP=8+SP, mslpr=8192. Each sample is 16 MiB and PP rank 0 needs 78 per bank. That holds ~8.5 GiB against the ~1.2 GiB that survives: about 7 GiB of capture-only peak, and 4x that at TP=2. Capture can OOM in a run whose steady state fits.
Suggestion: Allocate the leading samples once, for the first bank. Reuse the same tensor for the matching (chunk, slot, layer) in every other bank, and regenerate only the CP-dependent keyword inputs. Alternatively, drop the non-canonical leading inputs before calling TE.
There was a problem hiding this comment.
Agreed, fixed in 6bbffab with your second option. Each later bank reuses the first bank's leading hidden_states samples through TECudaGraphHelper._reuse_dynamic_cp_leading_samples, which checks that layouts and aliases match. CP-dependent args stay per bank, so only one sample set is live during warmup and capture. test_thd_cuda_graph.py::TestDynamicMicrobatchSlots::test_dynamic_cp_capture_reuses_leading_samples_across_banks covers the reuse, and the multi-bank replay tests still match eager. Capture peak memory at your scale is not measured yet.
| if scheduler_type == 'dp_balanced' | ||
| else getattr(config, 'thd_max_packed_sequences', None) | ||
| ) | ||
| scheduler_max_num_seqs = _get_scheduler_max_real_num_seqs(config) |
There was a problem hiding this comment.
[SUGGESTION Documentation] With this change every packing scheduler, not only dp_balanced, gets thd_max_packed_sequences - 1 when a dummy tail is appended. DefaultDynamicCPScheduler now enforces that cap per CP subgroup (line 467; data_schedule_utils.py:873-877, :1013-1015); on the base, DCP ignored it.
The field docstring, which is also the --thd-max-packed-sequences help text (transformer_config.py:1531-1540), was not updated:
- It still says "The dp_balanced packing scheduler reserves that dummy slot".
- Its sizing note ("setting it too large just allocates a slightly larger cu_seqlens buffer") no longer covers dynamic CP. With TE graphs,
train()reuses the value asthd_max_subsamples_per_itemfor non-varlen data (training.py:7895-7897). With PP>1, that sets the packed-microbatch range_validate_cuda_graph_microbatch_countenforces at runtime.
A DCP user tuning this for eager packing now changes both the pack cap and, with graphs, the accepted microbatch range.
Suggestion: Update the docstring: both schedulers cap real sequences per pack at this value (minus one with dummy-tail padding), applied per CP subgroup under dynamic CP. While train() still derives the per-item bound from this field, also say that with dynamic-CP CUDA graphs it bounds the accepted packed microbatch count.
There was a problem hiding this comment.
Agreed, updated in 6bbffab. The docstring and the --thd-max-packed-sequences help now say that both dp_balanced and default_dynamic_cp cap real sequences at this value (minus one with a dummy tail), and that dynamic CP applies the cap per CP subgroup. The field no longer sets the captured graph range (see the source-subsample thread).
Dynamic context parallelism (DCP) picks a CP size for every microbatch, and CP communicators are capture-time constants. TECudaGraphHelper therefore captures one graph bank per power-of-two CP size, from --min-dynamic-context-parallel-size to DP x CP, each with the real process group of that size. All banks share one TE memory pool through make_graphed_callables(_graph_memory_slots=...) (NVIDIA/TransformerEngine#3353), so graph memory does not grow with the number of banks. - Slot plan: the physical slot ring is sized from the liveness union of every PP/VPP order the DCP scheduler can produce. Saved-tensor and returned-gradient arenas are colored so that frames that are live at the same time never share storage. - Replay: a layer selects its bank (graphs, process group, static inputs) from PackedSeqParams.local_cp_size, including the mHC attention split and the hybrid wrapper, and checks that the microbatch's runtime CP group is the one the bank captured. A CP-off microbatch may carry its singleton group in cp_group or in cp_singleton_group. Router-only captures get a DCP PackedSeqParams so they use the bank's group. - Capture state: THD capture seeds a bounded RoPE table and cu_seqlens per bank, and eager RoPE uses the same bound under DCP. During training the packed microbatch count is checked against the captured range, and the DCP scheduler honors thd_max_packed_sequences, so a pack never exceeds the captured cu_seqlens capacity. Capture can be retried, and capture-only state is released on failure and on delete. - Config rules: DCP with CUDA graphs requires TE graphs and power-of-two DP x CP, and rejects delayed wgrad, EP overlap, moe_paged_stash and mamba scopes. Runs without DCP keep dev behavior apart from fixes on the paths they share: - The metadata-only THD padding path (PP stages without tokens) derives its padding target from the CP-local length, as the tensor path already does. - With CP=1 the dp_balanced scheduler gathers on the DPxCP communicator, which has the same ranks in the same order as DP. - Packed THD capture with fixed CP>1 seeds the full RoPE table and graph cu_seqlens instead of the CP-sliced table. - After an attention-only graph replay, the eager MLP receives the real PackedSeqParams. - PP stages without graphable layers still join the all-reduce that sizes the dynamic microbatch slot ring, so stages with graphs do not wait for them. Co-authored-by: HaochenYuan <haocheny@nvidia.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Graph slot memory protects only tensors saved for backward. With dynamic-CP graph banks, a replay of another bank can overwrite tensors that a forward kept in ctx attributes before the matching backward runs. The CSA sparse attention functions, the cuDNN sparse attention functions, FusedDSAIndexerLoss and the fused MLA RoPE in-place, concat and KV-split functions now pass that metadata through save_for_backward. Dynamic-CP capture walks several CP banks in sequence. It relies on CSA capture reusing any warmed-up compact-indexer workspace that matches the geometry (NVIDIA#8104); the tests cover this for BSHD and THD. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Review follow-ups for the dynamic-CP graph banks, and fine-grained activation offloading and FP8 attention inside captured scopes. - RoPE tables: a table read inside a graph must outlive the graph, but the class-wide LRU cache can evict it. While TE captures the banks, every RotaryEmbedding keeps the tables its warmups compute in a per-module cache; stream capture only reads that cache (a miss raises), and the tables are released when the graphs are deleted or capture aborts. Local graphs keep dev's capture-constant ownership. A DSA indexer graph test replays correctly after more than 32 other tables evict its table from the shared cache. - Capture memory: banks after the first reuse the first bank's leading hidden-state samples when their layouts and alias topology match, instead of keeping one set per bank until make_graphed_callables returns. - Microbatch bound: the capture range for real SFT items is derived from seq_length and the dataset's CP/SP padding divisor, independently of thd_max_packed_sequences, which only caps output packs. - Offloading: offload scopes inside a captured slot graph hand their saves to TE's slot hooks (_graph_memory_offload_context) instead of pushing the offload manager's own saved-tensor hooks inside TE's. The policy is frozen per module scope from eager warmup; attention, MLA, DSv4, experts and hybrid layers pass their module as the scope owner. - FP8 attention: TE#3353 now saves quantizer scale snapshots through autograd, so the dynamic-CP graph tests also cover fp8_dot_product_attention. - Docs, typing and the thd_max_packed_sequences description follow the new behavior. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
The style guide asks for double quotes in new code. Convert the single-quoted literals on lines this PR adds; lines moved unchanged from dev keep their quoting. No behavior change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Dynamic-CP graphs bound the RoPE table to the captured length while the packed max_seqlen can include a padding tail. apply_rotary_pos_emb then skips TE's fused THD RoPE, which needs the table to cover max_seqlen, and uses the unfused THD path. Replay that fallback forward and backward byte for byte for CP1 and both CP2 ranks, with a sequence that runs past the table, and note it in the rope_dispatch manifest entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
…undary - With pipeline parallelism, dynamic-CP graphs size their schedule from a bound on the sequences in one dataset item. GPT items with --dataloader-inter-document-masking can split into up to seq_length documents, so argument validation now rejects that combination instead of failing a valid step later. SFT keeps its derived bound, and varlen and FIM items hold one sequence. The SFT bound's comment now explains the +1. - Fine-grained activation offloading with whole-layer TE capture does not install the per-module offload boundary, which only had a warning. Dynamic-CP graphs now reject it and require partial cuda_graph_modules scopes. - The context-parallel and offloading guides state both limits, and the offloading guide describes the snapshot copies of asynchronous D2H. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
Capture built a PP/VPP order for every microbatch count up to the scheduler bound and colored all of them, so its cost grew quadratically with the bound. The SFT source-item bound can make that tens of thousands of microbatches, which would stall capture. Longer schedules only repeat the live sets of shorter ones: a reachable count's schedule table is a prefix of a longer one's, and past the warmup each forward/backward pair recurs on the same model chunks and physical slots every lcm(slots, VPP group size) microbatches. The proof now enumerates counts up to the warmup plus that period: first to size the slot ring, then, after the PP all-reduce, to color the ring's frames. The runtime contract still covers every count up to the scheduler bound. The slot-color test proves over the truncated counts, as capture does, and replays every reachable count up to twice the largest limit with the independent overlap check, now also with a VPP group size of twice the PP size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tailai Ma <tailaim@nvidia.com>
|
Re the [P2] in the review body (short-table THD RoPE determinism test): Thanks, added in 2c0fd65: We keep |
What does this PR do?
Lets dynamic context parallelism (DCP) use layer-wise Transformer Engine CUDA graphs. Each CP size gets its own graph bank, and all banks share one TE memory pool. FP8 attention and fine-grained activation offloading also work inside these graphs. DSv4/MLA backward metadata moves into saved tensors, the only state that shared graph memory protects.
Depends on NVIDIA/TransformerEngine#3353 as a whole (head
15fe37e0), not only itsmake_graphed_callables(_graph_memory_slots=...)argument. These graphs also rely on its slot offload (_graph_memory_offload_context), on the FP8 attention scales and THD all-gather metadata it saves through autograd, and on its physical-buffer saves of quantized tensors under CPU offload v1. Megatron checks only for_graph_memory_slots(DCP + CUDA graphs raises at construction without it) and_graph_memory_offload_context(graph offload raises at capture without it). Runs without DCP graphs do not need the TE change. Packed Gated DeltaNet capture (Qwen3.5) is left to a follow-up PR.Based on dev
43a2246f4.Design
TECudaGraphHelpercaptures every power-of-two CP size from--min-dynamic-context-parallel-sizeto DP×CP with that size's runtime group ([DCP V2 1/4] Define the runtime dynamic-CP group contract #7045). At replay, a layer picks its bank fromPackedSeqParams.local_cp_sizeand checks the microbatch's CP group against it.cu_seqlensper bank. While TE captures, eachRotaryEmbeddingkeeps the tables it returns in a per-module dict until the graphs are deleted, so a table built in warmup is still there at stream capture; a miss during stream capture raises instead of allocating the table in the graph pool. Banks share the leading tensor of each capture input after checking that layouts and aliasing match. The DCP scheduler caps each output pack atthd_max_packed_sequences. The capture bound counts source-item subsamples separately (SFT:seq_length // pad_divisor + 1; varlen, mock and other sources: 1), and each step's microbatch count is checked against it. The slot liveness proof enumerates only the microbatch counts up to the schedule's warmup pluslcm(slots, VPP group size), because longer schedules repeat their live sets, so planning cost does not grow with that bound.fp8_dot_product_attentionandfp8_multi_head_attention. TE#3353 saves the FP8 scale snapshots that backward reads, and the per-step THD all-gather metadata, through autograd, so the slot arenas protect them like any other saved tensor.save_for_backwardinstead ofctxattributes. Eager numerics are unchanged.Changes for runs without DCP (shared paths):
dp_balancedscheduler gathers on the equivalent DP×CP group.cu_seqlens.PackedSeqParamsto the eager MLP.delete_cuda_graphs()no longer asserts that graphs exist.max_seqlentakes the unfused path. TE's fused THD RoPE reads the table up tomax_seqlen.Review follow-up
cp_comm_type="all_gather".ctxattribute, so FP8 attention stays enabled.thd_max_packed_sequences. With PP,--dataloader-inter-document-maskingis rejected because its items have no such bound. Thethd_max_packed_sequencesdocstring describes the DCP cap.-> Noneon the capture wrapper, double quotes on added string literals.Commits
f8aa5d759Share TE CUDA graph memory across dynamic-CP graph banksbace48590Keep DSv4 and MLA RoPE backward metadata in saved tensors8bd9e61e4Fix dynamic-CP graph capture safety and offload compatibility (slot offload, per-module RoPE tables, SFT source bound, shared leading inputs)8ecc7c0b6Use double quotes for string literals added by dynamic-CP graphsf5fa8b50fAdd a determinism replay for the short THD RoPE table fallbackf689dd1f0Reject dynamic-CP graph inputs without a schedule bound or offload boundary9da6607ddBound the dynamic-CP slot liveness proof by the schedule periodResults (earlier revision, needs rerun)
Measured on
cdfa7c212(deva5aa637fa) with TE#33538980cd09. That run predates the rebase, FP8/offload support and the review fixes, and has not been rerun on9da6607dd. FP8 attention and offloading have no end-to-end training numbers.Qwen3-30B-A3B (48 layers, random init) on 8×GB200, TP1 PP4 VPP4 EP2, packed variable-length sequences, GBS 16,
--deterministic-mode, 50 steps. All runs use the same initial weights and data.attn moe_router moe_preprocess)attn)Bit-identical: LM loss, grad norm and MoE auxiliary loss at every step, and the final weights. Both CP banks are replayed (28 of 50 steps mix CP 1 and CP 2).
Tests
tests/unit_tests/transformer/test_thd_cuda_graph.py: slot plan and liveness colors (property test over PP 2/4/8 × VPP 1/2/4 with VPP group size PP or 2×PP: slots and colors are proved over the truncated counts as in capture, then every reachable count up to twice the limit is replayed with an independent overlap check), capture bounds against scheduler output, bank contexts, capture retry and delete, THD RoPE bounds, leading-input reuse across banks, FP8/FP4 config acceptance.tests/unit_tests/transformer/test_transformer_layer.py,test_mhc_attention_graph_split.py,test_cuda_graphs.py,test_vision_cuda_graphs.py: bank selection at replay, mHC attention split, slot-aliased variant order, delete before create.tests/unit_tests/transformer/test_hybrid_dcp_cuda_graph.py: hybrid mHC + MoE replay vs. eager across banks.tests/unit_tests/transformer/test_dcp_cuda_graph_offloading.py(new): offload event boundary, frozen scope policy, and a 4-rank PP2×DP2 run that compares losses and parameter gradients with eager for BF16/FP8 × dense/MoE × PP2/PP2-VPP2. It uses CP 1/2 banks, physical-slot wrap, microbatch counts 12 → 8 → 12 andcore_attn/attn_projoffload.tests/unit_tests/transformer/test_rope.py: RoPE and YaRN tables beyond the 32-entry cache survive capture.tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_dsa.py: aDSAIndexercaptured throughTECudaGraphHelperreplays like eager after 40 other tables evict its RoPE table from the shared cache and the freed memory is overwritten.tests/unit_tests/test_sequence_packing.py,tests/unit_tests/training/test_train_step_schedule_plumbing.py: microbatch-count check, DP×CP gather group, sample regrouping,train()helper construction from parsed defaults, SFT source-subsample bound.tests/unit_tests/transformer/test_transformer_config.py,tests/unit_tests/test_hybrid_context_parallel_args.py: dynamic-CP config and argument rules, including whole-layer capture + offload and PP + inter-document masking.tests/unit_tests/fusions/test_fused_mrope.py,tests/unit_tests/fusions/test_mla_yarn_rope_apply.py,tests/unit_tests/transformer/experimental_attention_variant/test_attention_variant_csa.py,test_csa_fused_sparse_attention_dev.py,test_attention_variant_dsa.py: short THD RoPE fallback, DSv4/CSA/DSA/MLA saved-tensor paths, rejection oflocalgraphs for packed DSA under DCP.tests/unit_tests/determinism/kernels/test_te_wrappers.py(+manifest.py): bit-exact forward/backward replay of the short-table THD RoPE fallback for CP1 and both CP2 ranks.TE-side tests (
tests/pytorch/test_cuda_graphs.py,tests/pytorch/attention/test_saved_quantizer_scales.py) are in NVIDIA/TransformerEngine#3353.Validation on
9da6607dd+ TE#335315fe37e0NGC PyTorch 26.03 containers with TE#3353 built from
15fe37e0.test_thd_cuda_graph.py,test_rope.py,test_hybrid_dcp_cuda_graph.py,test_mhc_attention_graph_split.py,test_pos_embedding_cuda_graph.py,test_hybrid_context_parallel_args.py,test_train_step_schedule_plumbing.py(4 ranks, one process per rank)test_dcp_cuda_graph_offloading.pyPP2×DP2 matrix and offload tests (4 ranks)test_attention_variant_dsa.py::TestDSAIndexer::test_graph_replay_keeps_rope_table_beyond_lru_capacitytest_te_wrappers.py -k rope,test_manifest.pytest_transformer_config.py -k dynamic_cp/test_sequence_packing.py(8 ranks)43a2246f4launch_on_gb200testsLimitations
--cuda-graph-impl transformer_enginewithcuda_graph_dynamic_microbatches.full_iteration/localgraphs (including dev's experimental packed-DSA path),step_batch_size_schedule, delayed wgrad, EP overlap,moe_paged_stashand mamba scopes are rejected.--dataloader-inter-document-maskingis rejected for non-SFT data. Other non-SFT sources are assumed to carry one sequence per item; a step outside the captured range fails the runtime check. The SFT bound is conservative; a larger bound only widens the runtime range, not the capture-time liveness proof.fp8_recipe="delayed"); the TE tests also cover current scaling. MXFP8/FP4 attention passes config validation without numerical coverage here. FP8 THD tests need SM100+ and cuDNN ≥ 9.23, so they skip on Hopper. Megatron's DCP graph replay tests do not usecp_comm_type="all_gather"; all-gather replay is covered only by the TE tests.cuda_graph_modulesscopes; whole-layer capture with offloading is rejected. The frozen per-scope policy can offload less than eager. There is no ahead-of-use prefetch: H2D is joined before backward uses the tensor. Async D2H adds a transient compute-stream copy of each offloaded save within the forward graph. TE allocates the host arenas outsidefine_grained_offloading_buffer_size_gib. Tested with GPT attention scopes only; MLA and hybrid offload under DCP graphs are untested.NVTE_CPU_OFFLOAD_V1=1is required, as for fine-grained offloading in general.WORLD_SIZE=4(FP8 cases also need SM100): the 8-rank H100 unit-test jobs skip it, and the GB200 jobs run onlylaunch_on_gb200tests.Determinism
The
determinism-exemptlabel covers five registered kernels that the coverage check still lists: fused MLA YaRN RoPE, attention dispatch, DSv4 q RMSNorm, DSA dispatch and MoE experts. Their edits move backward metadata intosave_for_backwardor pass the owning module to the offload interface; they do not change arithmetic or kernel selection. The fused DSA indexer-loss backward is not run by the registered DSA determinism tests;test_attention_variant_dsa.py::TestFusedDSAIndexerLossGradientchecks it against autograd. The one dispatch change that does (the short-table THD RoPE fallback) has its own bit-exact replay intest_te_wrappers.py.Issue tracking
Linked issue: none.
Contribution process
Pre-checks
🤖 Generated with Claude Code