Repository navigation
Conversation
With dynamic context parallelism, every training step repacks the global batch on the CPU before forward-backward starts. For each remaining sequence, next_dynamic_cp_group_packing_aware scored every group with room by taking the projected maximum over all DPxCP ranks, so the cost grew with sequences x groups x ranks: with mostly short sequences it took about 0.9 s per step at 128 DPxCP ranks and 4.7 s at 256. The ranks of a group, and all free ranks, always share one exec time, and costs are non-negative, so a placement's projected maximum is max(member time + cost, current maximum). Compute the current maximum and the free ranks once per sequence, and visit only the groups of the CP size being tried. Schedules and exec times are unchanged. AI assistance (Cursor) was used for this change. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: YangHaoooooo <233765393+Hang-get@users.noreply.github.com>
Hang-get
marked this pull request as ready for review
November 11, 2026 08:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Reduces the CPU time of the dynamic-CP packing scheduler, which runs before every training step, by scoring each candidate placement in constant time instead of scanning every DP×CP rank. Schedules are unchanged.
Issue tracking
Linked issue: N/A (small performance improvement)
Problem
With
--dynamic-context-parallel(schedulerdefault_dynamic_cp),wrap_data_iterator()repacks the global batch at the start of every training step:DefaultDynamicCPScheduler.get_groups_and_subsamples()callsnext_dynamic_cp_group_packing_aware()once per packed microbatch until every sequence is placed. It runs on every TP rank 0 after the data-exhaustion check has synchronized with the device and before forward-backward is launched, and the other TP ranks wait for its result, so the whole step waits for it.For each remaining sequence and each CP size, the function walked every group and, for every group of that size with room (and for the new-group candidate), built a member set and took the projected maximum exec time over all DP×CP ranks. Sequences that do not fit are scored again in every call, so the cost grows roughly with sequences × groups × ranks. With mostly short sequences, which share many CP1 groups, it takes about 0.9 s per step at 128 DP×CP ranks and 4.7 s at 256 in the benchmark below.
Change
In the main loop all ranks of a group have the same exec time (a group starts from free ranks, and every placement adds the same cost to each member), free ranks stay at zero, and costs are non-negative. The projected maximum of a placement is therefore
max(member time + cost, current maximum): if the members are below the current maximum, another rank holds it; if they hold it, adding a non-negative cost keeps them at or above every other rank.The current maximum and the sorted free ranks are now computed once per sequence, the per-rank length once per CP size, and each CP size visits only the groups of that size, kept in creation order (the order in which
group_sizewas iterated). Candidate scores, tie-breaking and the returned schedules and exec times are unchanged.Benchmark
CPU benchmark of the real
DefaultDynamicCPScheduler.get_groups_and_subsamples()(pure Python) on an Intel Xeon Platinum 8352Y, one thread, CPython 3.12. Synthetic global batches: 90% of the sequences have log-normal lengths (median about 1.8k tokens) and the rest are uniform between 16k and 128k tokens;max_seqlen_per_dp_cp_rank=16384,min_dynamic_context_parallel_size=1. Each value is the median over 3 seeds of the median of 3 alternating runs;mainand this PR returned identical schedules for every input.mainWith fewer and longer sequences (half of them short, 4 per rank) the scheduler is cheaper and the gain smaller: 118 -> 31 ms (3.8x) at 128 ranks and 515 -> 100 ms (5.1x) at 256; at 32 and 64 ranks it saves 4 and 15 ms (1.9x and 2.6x).
Tests
test_next_dynamic_cp_group_packing_aware_scores_placements_by_busiest_rankintests/unit_tests/test_sequence_packing.py: two hand-checked schedules, one where member exec times decide the group and one where another group sets the maximum and ties go to the first candidate. It passes onmain, and it fails if a placement is scored without the current maximum or without the member time.python -m pytest -q tests/unit_tests/test_sequence_packing.py tests/unit_tests/data/test_packed_schedule_regressions.py— 120 passed, 1 skipped, 27 failed.maingives the same result apart from the new test (119 passed). All 27 failures areRuntimeError: Distributed package doesn't have NCCL built in(CPU build of PyTorch).mainimplementation ofnext_dynamic_cp_group_packing_awareover 30,000 cases (646,953 calls, 5,312 of them raising) found no difference: DP×CP sizes from 1 to 128 including 3, 6, 12, 14, 24, 48 and 96, random subsets of CP sizes andmin_cp_size, padding on and off, equal lengths, and sequences that fit no group. It repeats the scheduler's leftover loop and compares all four return values, including exec times, and the raised errors exactly.BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh— Black, isort, Pylint and Ruff passed; the non-gating mypy step reports the same four existing findings as onmain.All tests ran on CPU. GPU unit tests and the
hybrid_dynamic_cp_*functional tests were not run.Related open work on
dev: draft #8495 rewrites thedevcopy of this candidate loop (placements are scored bycandidate_score, which adds a communication term, and--dynamic-cp-runtime-configcan move planning one batch ahead into a CPU worker), and #8513 and #6479 add conditions inside it (max_cp_size,max_num_seqs). This PR changes only the synchronous scheduler onmain; syncing the branches would need a manual merge of this loop with #8495.Code provenance
No code was copied or adapted from another repository.
Contribution process
Pre-checks
jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it intests/unit_tests/determinism/kernels/manifest.py(guide)tools/autoformat.shon my PRAI assistance (Cursor) was used for this change.