Skip to content

Speed up dynamic CP microbatch scheduling - #9346

Open
Hang-get wants to merge 1 commit into
NVIDIA:mainfrom
Hang-get:hang-get/perf-dynamic-cp-scheduling
Open

Hang-get wants to merge 1 commit into
NVIDIA:mainfrom
Hang-get:hang-get/perf-dynamic-cp-scheduling

Conversation

@Hang-get

@Hang-get Hang-get commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Reduces the CPU time of the dynamic-CP packing scheduler, which runs before every training step, by scoring each candidate placement in constant time instead of scanning every DP×CP rank. Schedules are unchanged.

Issue tracking

Linked issue: N/A (small performance improvement)

Problem

With --dynamic-context-parallel (scheduler default_dynamic_cp), wrap_data_iterator() repacks the global batch at the start of every training step: DefaultDynamicCPScheduler.get_groups_and_subsamples() calls next_dynamic_cp_group_packing_aware() once per packed microbatch until every sequence is placed. It runs on every TP rank 0 after the data-exhaustion check has synchronized with the device and before forward-backward is launched, and the other TP ranks wait for its result, so the whole step waits for it.

For each remaining sequence and each CP size, the function walked every group and, for every group of that size with room (and for the new-group candidate), built a member set and took the projected maximum exec time over all DP×CP ranks. Sequences that do not fit are scored again in every call, so the cost grows roughly with sequences × groups × ranks. With mostly short sequences, which share many CP1 groups, it takes about 0.9 s per step at 128 DP×CP ranks and 4.7 s at 256 in the benchmark below.

Change

In the main loop all ranks of a group have the same exec time (a group starts from free ranks, and every placement adds the same cost to each member), free ranks stay at zero, and costs are non-negative. The projected maximum of a placement is therefore max(member time + cost, current maximum): if the members are below the current maximum, another rank holds it; if they hold it, adding a non-negative cost keeps them at or above every other rank.

The current maximum and the sorted free ranks are now computed once per sequence, the per-rank length once per CP size, and each CP size visits only the groups of that size, kept in creation order (the order in which group_size was iterated). Candidate scores, tie-breaking and the returned schedules and exec times are unchanged.

Benchmark

CPU benchmark of the real DefaultDynamicCPScheduler.get_groups_and_subsamples() (pure Python) on an Intel Xeon Platinum 8352Y, one thread, CPython 3.12. Synthetic global batches: 90% of the sequences have log-normal lengths (median about 1.8k tokens) and the rest are uniform between 16k and 128k tokens; max_seqlen_per_dp_cp_rank=16384, min_dynamic_context_parallel_size=1. Each value is the median over 3 seeds of the median of 3 alternating runs; main and this PR returned identical schedules for every input.

DP×CP ranks Sequences per step Tokens per step main This PR Speedup
32 256 ~2.6M 19.8 ms 6.6 ms 3.0x
64 512 ~4.6M 144 ms 20.9 ms 6.9x
128 1,024 ~9.1M 891 ms 73.5 ms 12.1x
256 2,048 ~18.1M 4,742 ms 227 ms 20.9x

With fewer and longer sequences (half of them short, 4 per rank) the scheduler is cheaper and the gain smaller: 118 -> 31 ms (3.8x) at 128 ranks and 515 -> 100 ms (5.1x) at 256; at 32 and 64 ranks it saves 4 and 15 ms (1.9x and 2.6x).

Tests

  • New test_next_dynamic_cp_group_packing_aware_scores_placements_by_busiest_rank in tests/unit_tests/test_sequence_packing.py: two hand-checked schedules, one where member exec times decide the group and one where another group sets the maximum and ties go to the first candidate. It passes on main, and it fails if a placement is scored without the current maximum or without the member time.
  • python -m pytest -q tests/unit_tests/test_sequence_packing.py tests/unit_tests/data/test_packed_schedule_regressions.py — 120 passed, 1 skipped, 27 failed. main gives the same result apart from the new test (119 passed). All 27 failures are RuntimeError: Distributed package doesn't have NCCL built in (CPU build of PyTorch).
  • A randomized comparison with the main implementation of next_dynamic_cp_group_packing_aware over 30,000 cases (646,953 calls, 5,312 of them raising) found no difference: DP×CP sizes from 1 to 128 including 3, 6, 12, 14, 24, 48 and 96, random subsets of CP sizes and min_cp_size, padding on and off, equal lengths, and sequences that fit no group. It repeats the scheduler's leftover loop and compares all four return values, including exec times, and the raised errors exactly.
  • BASE_REF=main CHECK_ONLY=true SKIP_DOCS=false bash tools/autoformat.sh — Black, isort, Pylint and Ruff passed; the non-gating mypy step reports the same four existing findings as on main.

All tests ran on CPU. GPU unit tests and the hybrid_dynamic_cp_* functional tests were not run.

Related open work on dev: draft #8495 rewrites the dev copy of this candidate loop (placements are scored by candidate_score, which adds a communication term, and --dynamic-cp-runtime-config can move planning one batch ahead into a CPU worker), and #8513 and #6479 add conditions inside it (max_cp_size, max_num_seqs). This PR changes only the synchronous scheduler on main; syncing the branches would need a manual merge of this loop with #8495.

Code provenance

No code was copied or adapted from another repository.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • If this PR adds or changes a GPU kernel (Triton, jit_fuser/torch.compile, CUDA extension, TE or external-library dispatch, or a scatter/index accumulation), I have added or updated its bit-exact determinism test and registered it in tests/unit_tests/determinism/kernels/manifest.py (guide)
  • I have added proper typing to my code
  • I have added relevant documentation
  • I have run tools/autoformat.sh on my PR

AI assistance (Cursor) was used for this change.

With dynamic context parallelism, every training step repacks the global
batch on the CPU before forward-backward starts. For each remaining
sequence, next_dynamic_cp_group_packing_aware scored every group with
room by taking the projected maximum over all DPxCP ranks, so the cost
grew with sequences x groups x ranks: with mostly short sequences it
took about 0.9 s per step at 128 DPxCP ranks and 4.7 s at 256.

The ranks of a group, and all free ranks, always share one exec time,
and costs are non-negative, so a placement's projected maximum is
max(member time + cost, current maximum). Compute the current maximum
and the free ranks once per sequence, and visit only the groups of the
CP size being tried. Schedules and exec times are unchanged.

AI assistance (Cursor) was used for this change.

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: YangHaoooooo <233765393+Hang-get@users.noreply.github.com>
@Hang-get
Hang-get marked this pull request as ready for review November 11, 2026 08:02
@Hang-get
Hang-get requested review from a team as code owners November 11, 2026 08:02
@copy-pr-bot

copy-pr-bot Bot commented Nov 11, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Nov 13, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants