Skip to content

fix: re-register a module's FP8 tensors after FP8GlobalStateManager.reset() - #3913

Draft
lxcxjxhx wants to merge 1 commit into
NVIDIA:mainfrom
lxcxjxhx:fix/fp8-reregister-after-reset
Draft

lxcxjxhx wants to merge 1 commit into
NVIDIA:mainfrom
lxcxjxhx:fix/fp8-reregister-after-reset

Conversation

@lxcxjxhx

Copy link
Copy Markdown
Contributor

Description

Fixes #3858.

FP8GlobalStateManager.reset() replaces the process-global state, but a module's registration marker (fp8_meta["buffer_index_and_autocast_key"]) is stored on the module, so it survives the reset. add_fp8_tensors_to_global_buffer() only tested for the marker's presence, so a module reused after a reset never registered into the replacement global_amax_buffer. Its amax was still collected locally while nothing reached reduce_and_update_fp8_tensors(), and its delayed-scaling scale stopped adapting: with weights of 1/16 and sixteen inputs of 128, the output stayed at 1.0 instead of 128.0 on every step.

The state now carries a generation that reset() increments, and a marker is only trusted while it belongs to the current generation:

if (
    fp8_meta.get(index_in_buffer) is not None
    and fp8_meta.get("buffer_generation") == qstate.buffer_generation
):
    return

Otherwise the stale marker is cleared and the module registers again with the new buffers, which is the semantics the existing comment already claimed ("exactly once per global state").

Two consequences worth naming:

  • buffer_generation is bookkeeping, not FP8 tensor state, so it is excluded from the pickled extra state alongside the registration marker. FP8GlobalStateManager.BOOKKEEPING_KEYS is now the single place listing those keys, used by get_extra_state in module/base.py and ops/op.py. Without that, checkpoints would grow an internal key whose value is meaningless in the process that loads them.
  • Saving and loading is unaffected in the other direction: a checkpoint has no marker, so a loaded module registers normally, exactly as before.

Testing

What I ran

  • black --line-length 100 --check with the version the repository pins (24.4.2) on all four touched files: unchanged. (Black 26 wants an unrelated blank line after the module docstring; that reformat is deliberately not included.)
  • python -m py_compile on the touched files.
  • pylint==3.3.1 on transformer_engine/pytorch/quantization.py: 9.93/10, the only message being E0611 for transformer_engine.common, an artefact of not having the extension installed here.

What I did not run

This host has no CUDA toolkit and no built transformer_engine_torch, so the new test and the reporter's reproduction script could not be executed here. The PR is opened as a draft for that reason, and per AGENTS.md it should stay a draft until its human author has reviewed it. A /te-ci run is what would confirm it.

The test (tests/pytorch/test_sanity.py::test_delayed_scaling_adapts_after_global_state_reset) follows the reporter's sequence with fewer moving parts: warm up at amplitude 1, reset(), assert the global buffer is empty, run one step to force re-registration, then two steps at amplitude 128. It asserts 2 registered entries after the reuse (the failure mode is 0), that the first step after the jump still uses the stale scale, and that the second one reaches the reference mean of 128. Before the change the same sequence reports 0, 0, 0 entries and an output mean of 1 for every step, which is the table in #3858.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation change
  • New feature
  • Performance improvement
  • Tests

Checklist

  • I have read and followed the Contributing Guidelines.
  • The changes are formatted with black at line length 100 and pass the touched-file lint.
  • There are no new warnings from the changes.
  • The commit is signed off (git commit -s).
  • qa/L0_pytorch_unittest/test.sh and the L0_* checks have been run — not possible on this host (no GPU, no built extension); this needs the /te-ci run.

Additional information

Scope is limited to the registration guard and the two places that enumerate fp8_meta keys for serialization. init_fp8_metadata and the recipe-state reuse path are untouched: an existing module keeps its own scale and amax tensors, which is what makes reuse meaningful.

…eset()

Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PyTorch] Reusing a DelayedScaling module after FP8GlobalStateManager.reset() leaves stale registration and scales

1 participant