Skip to content

Use HybridAdam for Gemini CPU optimizer offload - #6435

Open
sdjasj wants to merge 1 commit into
hpcaitech:mainfrom
sdjasj:agent/fix-gemini-fused-adam-offload-6388
Open

Use HybridAdam for Gemini CPU optimizer offload#6435
sdjasj wants to merge 1 commit into
hpcaitech:mainfrom
sdjasj:agent/fix-gemini-fused-adam-offload-6388

Conversation

@sdjasj

@sdjasj sdjasj commented Jul 29, 2026

Copy link
Copy Markdown

What changed

  • detect when Gemini static placement actually assigns one or more optimizer shards to CPU
  • automatically replace FusedAdam with HybridAdam in that case while preserving the original parameter groups and their hyperparameters
  • add a two-step distributed regression using the reported offload_optim_frac=1.0 and min_chunk_size_m=2 settings

Why

FusedAdam only has a CUDA update kernel, but Gemini listed it as supported even when static placement moved master shards, gradients, and optimizer state to CPU. Small chunk sizes make those shards eligible for offload, and the fused CUDA kernel then failed with expected input to be on cuda.

HybridAdam dispatches each update to the CPU or CUDA implementation according to the parameter device and is the compatible implementation for mixed Gemini placement.

Impact

Users can keep constructing FusedAdam while using Gemini optimizer offload; Gemini selects HybridAdam only when CPU shards are actually present.

Fixes #6388.

Validation

  • python -m pytest tests/test_zero/test_gemini/test_fused_adam_offload.py -q (2 GPUs; 1 passed)
  • the regression runs two consecutive forward/backward/optimizer steps and asserts CPU placement plus the optimizer conversion
  • compileall and git diff --check: passed

@sdjasj
sdjasj marked this pull request as ready for review August 1, 2026 12:03
@sdjasj
sdjasj requested a review from a team as a code owner August 1, 2026 12:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: GeminiPlugin with offload_optim_frac=1.0 and small min_chunk_size_m causes RuntimeError: expected input to be on cuda during training

1 participant