Skip to content

fix(mistral): freeze text-only visual branches and enable resume - #3964

Merged
yuhezhang-ai merged 1 commit into
mainfrom
yuhez/fix/amint-9-text-only-freeze
Sep 21, 2026
Merged

yuhezhang-ai merged 1 commit into
mainfrom
yuhez/fix/amint-9-text-only-freeze

Conversation

@yuhezhang-ai

@yuhezhang-ai yuhezhang-ai commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

The Ministral3 SQuAD recipes adapt only the language model, but full fine-tuning still includes the unused vision tower and its sibling multimodal projector in Adam. Freeze both explicitly with the existing module-selector config and re-enable checkpoint-resume validation for both full and PEFT training.

PR #3635 already materializes missing Adam state before saving new checkpoints. Explicit freezing additionally keeps permanently inactive visual parameters out of the optimizer. PEFT already freezes these branches through its base-model policy and LoRA exclusions; the explicit selectors document and enforce the text-only recipe policy in both examples.

Changelog

  • Set freeze_config.freeze_modules to model.vision_tower and model.multi_modal_projector in both Ministral3 SQuAD recipes.
  • Remove both ci.checkpoint_robustness.skip_resume exemptions and their stale workaround comments.
  • Add a tiny real-VLM regression test that reads each recipe and verifies frozen visual parameters, exclusion from optimizer groups, and retained trainable language/LoRA parameters.

Validation

  • 48 focused CPU freeze/selector/optimizer-state tests passed.
  • Four tiny-VLM CPU strict DCP optimizer round trips passed: full and PEFT, each with and without explicit freezing. Restored optimizer state and the next update matched exactly. This does not substitute for distributed 3B recipe coverage.
  • Repository Ruff formatting/lint and git diff --check passed.
  • Generated CI definitions include resume for both recipes and retain cross_tp_reload for full fine-tuning.
  • Scoped EOS CI passed for commit df3476d0f5c05a1fbe407b5371b14790aef71e24: pipeline 69061370. Both recipes ran on 8 H100 GPUs with the shipped CI overrides and resume enabled.
Recipe Source reference/parity Train/save AutoModel/HF reload Resume Cross-TP reload
Full PASS PASS PASS PASS (67 s) PASS
PEFT PASS PASS PASS PASS (76 s) Not configured

Before this PR is ready for review

  • Read and followed the contributor guidelines; commit includes DCO signoff.
  • Added the necessary focused tests.
  • Documented language-only adaptation in the recipe comments.
  • Full and PEFT scoped checkpoint-robustness CI passes, including resume.

Related to AMINT-9.

Signed-off-by: Yuhe Zhang <yuhez@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yuhezhang-ai

Copy link
Copy Markdown
Contributor Author

/ok to test df3476d

@athitten athitten left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @yuhezhang-ai !

@yuhezhang-ai
yuhezhang-ai merged commit 4702153 into main Sep 21, 2026
87 checks passed
@yuhezhang-ai
yuhezhang-ai deleted the yuhez/fix/amint-9-text-only-freeze branch September 21, 2026 19:39

This branch was successfully deployed

3 active deployments
public — df3476d0 Deployed Sep 21, 2026 by copy-pr-bot[bot] via release / finalize / notify #4718
test — df3476d0 Deployed Sep 21, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #10889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants