Skip to content

[v5] Fix checkpoint writes and RL logging - #348

Merged
tastelikefeet merged 5 commits into
dev-swift-v5from
fix/v5-bug-1028
Oct 29, 2026
Merged

tastelikefeet merged 5 commits into
dev-swift-v5from
fix/v5-bug-1028

Conversation

@qychen2001

@qychen2001 qychen2001 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Problem and changes

修复 v5 RL 的 checkpoint 并发写入与逐样本日志,增加 worker 实际精度诊断。目标分支为 dev-swift-v5。

  • Transformers save_pretrained 的 is_main_process=False 不能阻止当前版本写权重分片。现在完整模型仅由主 rank 写权重,所有 rank 仍参与模型/优化器状态收集;覆盖同步与异步保存。这不修复提前写入 trainer_state 完成标记的问题。
  • 回复日志在 prompt 缺失时从 messages 补充题目,保留 messages、reasoning_content、completion_length、truncated。评分不变,不把 reasoning 当作最终答案。
  • FSDP worker 构造策略时记录参数存储 dtype、mixed_precision 及完整 MixedPrecisionPolicy。关闭 BF16 的根因在配套 Swift builder 中修复,本仓库不改已有 BF16 policy 构造逻辑。

配套 Swift PR:modelscope/ms-swift#10451 (sampler 放置、共置日志调用、RL-04 精度与 RL-14 shuffle)。

Scope and compatibility

按用户要求,已通过 revert 完整撤回 optimizer_group 的 GA 更新边界、公共指标编号、Transformers/Megatron 双 RPC 事务指标以及相关测试修改。恢复修改前行为,本 PR 不再声明修复 RL-01。Swift 的配套步数预算和边界修改也已撤回。

Validation

  • checkpoint writer CPU 测试覆盖主/非主 rank、同步/异步保存;completion logging 测试覆盖日志字段。
  • 历史 8 PPU、Qwen3.5-4B colocate 实验确认单 writer 权重分片可读取及逐样本日志生成。
  • 配套 Swift 精度/shuffle CPU 回归验证真实 FSDP policy;模型加载和远程构造为替身。
  • 回退文件与 GA 修改前逐一核对一致。
  • 此前 cur_step=12、4/8/12 更新和 96 completions 的实机结果依赖现已撤回的 GA 修改,不代表当前分支行为。早期配置为 BF16 的实机运行实际 mixed_precision=no,不能作为 RL-04 的 BF16 验收。
  • 新增精度诊断及本次回退后未重跑多卡训练,没有异构/异步、Megatron 或 checkpoint 续训的完整实机验收。
PYTHONPATH=src python tests/transformers/test_checkpoint_writer_rank.py -v
PYTHONPATH=src python tests/agentic/test_completion_logging.py -v

@qychen2001 qychen2001 changed the title [v5] Fix checkpoint writers, gradient accumulation and rollout logs [v5] Fix checkpoint writes, accumulation and RL diagnostics Oct 28, 2026
@qychen2001
qychen2001 marked this pull request as ready for review October 28, 2026 09:26
Comment thread src/twinkle/model/optimizer_group.py Outdated
status = self.train_status if is_training else self.eval_status
if metric_step is None:
metric_step = self.cur_step - 1
metric_step = self.cur_step

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

注意下这里是否是 v5 分支的修改,如果是 main 分支已有代码,看看这里原先的设计是否本就有意义

@qychen2001 qychen2001 changed the title [v5] Fix checkpoint writes, accumulation and RL diagnostics [v5] Fix checkpoint writes and RL logging Oct 28, 2026
@tastelikefeet
tastelikefeet merged commit 4e09904 into dev-swift-v5 Oct 29, 2026
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants