Skip to content

fix: honor explicit seed=0, un-dead the distributed resync branch, exception-safe _deterministic_seed#1917

Open
arham766 wants to merge 1 commit into
NVIDIA:mainfrom
arham766:fix/random-seed-handling
Open

fix: honor explicit seed=0, un-dead the distributed resync branch, exception-safe _deterministic_seed#1917
arham766 wants to merge 1 commit into
NVIDIA:mainfrom
arham766:fix/random-seed-handling

Conversation

@arham766

@arham766 arham766 commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Type of change: Bug fix

Three related defects in modelopt/torch/utils/random.py (found writing #1916, confirmed by two reviews): is_manual was computed after the seed was reassigned, so it was ALWAYS True and the distributed resync branch for auto-seeded generators was dead code (ranks could sample divergent AutoNAS subnets); the same falsy check silently replaced an explicit seed=0 with random bits; and _deterministic_seed leaked the seed-1024 generator (and both flags) if the body raised. Regression tests are in a fix-scoped file (test_random_seed.py) so they don't collide with #1916's fuller suite; each was verified to fail pre-fix with the exact symptom (seed-0 stream divergence, is_manual True, leaked generator).

Usage

N/A

Testing

Regression tests included, each verified to FAIL with the fix reverted. Combined battery with all sibling wave fixes: 381 passed across tests/unit/torch/opt, tests/unit/torch/utils, tests/unit/recipe, and quantization test_mode. Note: the unmerged suite PRs #1913-#1916 contain behavior-documenting NOTE tests pinning the OLD behavior fixed here — whichever lands second gets the NOTE tests flipped (happy to rebase either way).

Before your PR is "Ready for review"

  • Is this change backward compatible?: ✅
  • If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: N/A
  • Did you write any new necessary tests?: ✅
  • Did you update Changelog?: N/A
  • Did you get Claude approval on this PR?: N/A (external contributor)

Additional Information

Issue: #1902 (wave-2 findings)

Summary by CodeRabbit

  • Bug Fixes

    • Improved random seed handling so explicit seeds are preserved correctly, including 0.
    • Restored generator state more reliably after temporary deterministic operations, even when errors occur.
    • Kept additional generator metadata in sync when switching between seeded and generated random states.
  • Tests

    • Added regression coverage for seed consistency, manual-seed tracking, and state restoration after exceptions.

…ception-safe _deterministic_seed

Three related defects in the shared RNG helper:

- is_manual was computed AFTER seed was reassigned by
  dist.broadcast(seed or getrandbits(64)), so it was always True and
  the resync branch for auto-seeded generators created before
  process-group init was dead code - distributed ranks could sample
  divergently (e.g. AutoNAS RandomSearcher subnets).
- The same falsy check silently replaced an explicit seed=0 with
  random bits.
- _deterministic_seed yielded without try/finally, leaking the
  seed-1024 generator (and the is_manual/is_synced flags) if the body
  raised, e.g. a failing sample callback in nas algorithms; the
  restore now also covers both flags, matching the docstring's
  'resets the random state to prior upon exit'.

Regression tests live in a fix-scoped file; the fuller suite for this
module is proposed separately in NVIDIA#1916.

Part of the findings in NVIDIA#1902 (wave 2).

Signed-off-by: arham766 <arhamislam766@yahoo.com>
@arham766 arham766 requested review from a team as code owners July 6, 2026 01:49
@arham766 arham766 requested a review from realAsma July 6, 2026 01:49
@copy-pr-bot

copy-pr-bot Bot commented Jul 6, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0f3f275-d7ca-4f4c-8a56-582808631444

📥 Commits

Reviewing files that changed from the base of the PR and between b96a785 and d15828e.

📒 Files selected for processing (2)
  • modelopt/torch/utils/random.py
  • tests/unit/torch/utils/test_random_seed.py

📝 Walkthrough

Walkthrough

This PR updates _get_generator to track an is_manual flag reflecting whether a seed was explicitly provided, and modifies _deterministic_seed to snapshot and restore is_manual and is_synced state on context exit. New regression tests validate seed reproducibility, flag correctness, and state restoration after exceptions.

Changes

Random Generator Seed Handling

Layer / File(s) Summary
Generator initialization with is_manual tracking
modelopt/torch/utils/random.py
_get_generator now derives is_manual separately from seed broadcasting logic, generating a 64-bit seed when none is provided, and stores is_manual on the generator object.
Deterministic seed context restores flags
modelopt/torch/utils/random.py
_deterministic_seed captures prior is_manual and is_synced flags before setting a deterministic seed, and restores both flags along with the prior generator instance on exit.
Regression tests for seed and flag behavior
tests/unit/torch/utils/test_random_seed.py
New test module with an autouse fixture clearing cached generator state, and tests covering explicit seed reproducibility, is_manual flag correctness, and generator restoration after exceptions inside _deterministic_seed.

Estimated code review effort: 2 (Simple) | ~12 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant DeterministicSeedContext
  participant GetGenerator

  Caller->>DeterministicSeedContext: enter context(seed)
  DeterministicSeedContext->>GetGenerator: capture prior generator, is_manual, is_synced
  DeterministicSeedContext->>GetGenerator: set deterministic seed
  Caller->>Caller: execute context body
  DeterministicSeedContext->>GetGenerator: restore prior generator, is_manual, is_synced
Loading

Related PRs: None specified.

Suggested labels: bug, tests

Suggested reviewers: None specified.

🐰 A seed once lost is found again,
is_manual whispers where it began,
through context's rise and fall it flows,
the generator remembers, and onward goes.

🚥 Pre-merge checks | ✅ 6
✅ Passed checks (6 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the explicit-seed fix, distributed resync behavior, and exception-safe deterministic seeding.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Security Anti-Patterns ✅ Passed Changed modelopt files only adjust RNG state/tests; no torch.load(weights_only=False), numpy.load(allow_pickle=True), trust_remote_code=True, eval/exec, # nosec, or new deps found.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant