Skip to content

[None][fix] MiniMax-H3 static FP8 scenario: any sequence length, optional toolchain, GPU tests - #22719

Merged
karljang merged 5 commits into
NVIDIA:feat/minimax-h3-rubinfrom
karljang:fix/h3-fp8-scenario-followup
Nov 21, 2026
Merged

karljang merged 5 commits into
NVIDIA:feat/minimax-h3-rubinfrom
karljang:fix/h3-fp8-scenario-followup

Conversation

@karljang

@karljang karljang commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Follow-up to #22658 on feat/minimax-h3-rubin: adjustments to the MiniMax-H3 static FP8 scenario so it serves any request shape and depends on less at load.

  • Any packed sequence length. The Down and attention-output projections now share one kernel (kernels/projection_residual.py) that takes the row count at run time and is compiled once at load. The sequence-length allowlist and the separate fallback path are removed.
  • CUDA toolchain optional. Without nvcc, ninja or cuobjdump, or under python -O, the installed quantize and cast operators run instead of the conversion kernels built at load. The same happens when the build or the inspection of the built binaries fails. The output is identical; transformer.fp8_kernel_qualification reports which is active. A built kernel whose output differs from the installed operators still fails the load.
  • No projection kernel compiles in the denoise loop. They are compiled at load; an input outside their dtypes, layout or 256-byte alignment takes the unfused cuBLAS path, and the first such call of each projection logs a warning.
  • Failure-safe load. Kernels compile before GateUp weights are packed, and an already packed block is not packed again.
  • Route declines instead of failing at load when the shared scenario settings were already selected with other values in this process: auto keeps the family pipeline, require names the reason.
  • Less duplication. Q/K norm + RoPE reuses the NVFP4 scenario's fused_fp8_qk kernel through a rope_halves_equal option; the shared head cast goes through a small family hook. transformer_forward.py, kernels/rope.py and kernels/attention_output.py are removed (about 850 fewer lines net).
  • Memory. The conditioning cache is bounded (64 keys, least recently used evicted); packed GateUp weights replace the checkpoint layout in place instead of keeping a second copy.
  • Packaging. The scenario's .cu/.cpp sources are no longer dropped from wheels.
  • Hygiene. Bandit marker for the subprocess import (the pre-commit failure on [None][perf] Add native static FP8 H3 scenario on Rubin #22658); report links and attribution text removed from the README and kernel headers; README rewritten for the above.

Known limits, unchanged by this PR: the scenario still shares per-process attention and decoder settings with the NVFP4 scenario (and subclasses its model classes), and where the normalized hidden state is NaN the fused AdaLN kernel writes a saturated FP8 code rather than the FP8 NaN code.

Test Coverage

  • test_fp8_contracts.py (CPU, l0_cpu.yml): 32 tests, 75 subtests pass locally.
  • New test_fp8_kernels.py (SM107, added to l0_gr100.yml): each fused op against the unfused operators at six sequence lengths from 1 to 38169, counting calls into the fused kernel and the cuBLAS path; the unfused path against the fused result for misaligned, int32-index and FP32-modulation inputs; a per-element bound for GateUp+SwiGLU; non-finite values.
  • test_fp8_conversions.py (SM107 + toolchain, added to l0_gr100.yml): built conversions byte-identical to the installed operators, directly and through the ops (calls into the built modules are counted). Skips when the kernels cannot be built, so the CI image needs nvcc, ninja and cuobjdump for it to run.
  • test_h3_scenario_fused_fp8_qk.py (already in l0_gr100.yml): with a table whose halves differ, rope_halves_equal=True gives the codes of the first half repeated and the default does not, at the FP8 scenario's launch shape.
  • test_minimax_h3_attention_precision.py (CPU, already in l0_cpu.yml): a new test pins the family RoPE table's equal halves, which rope_halves_equal relies on.
  • The l0_gr100 entries run post-merge, so no check on this PR executes the GPU tests; the results below are from a manual run at this head.
  • test_fp8_pipeline.py now skips, rather than fails, without the checkpoint or NVIDIA_TF32_OVERRIDE=0. It is not registered in CI because it needs the FP8 checkpoint.

Verified on an SM107 GPU in the side-branch release image with this branch's Python files laid over it:

Check Result
test_fp8_kernels.py + test_fp8_conversions.py + test_h3_scenario_fused_fp8_qk.py, and the CPU RoPE-table test, one pytest process 97 passed
FP8 checkpoint, 768x1344x124, 50 steps, a length #22658 already fused: this PR vs #22658 video and audio bit-identical
Same request with the CUDA tools removed from PATH bit-identical to the run with tools
FP8 checkpoint, a length #22658 served through its fallback runs on the fused kernels; output differs from the fallback, as expected
NVFP4 scenario (shares the touched QK kernel and family hook): image vs #22658 vs this PR bit-identical

PR Checklist

  • The change follows the coding guidelines; pre-commit passes on the changed files.
  • Tests cover the change.
  • No public API change.

🤖 Generated with Claude Code

…n, clean docs

- The route declines (NoScenario) instead of failing at load when a build
  tool the conversion kernels need (nvcc, ninja, cuobjdump) is missing, or
  when the shared scenario settings are already selected with other values.
  With visual_gen_scenario=auto the family pipeline is kept in both cases.
- Mark the subprocess import for Bandit (B404); it runs cuobjdump on the
  kernels the module just built.
- README and kernel headers: drop report links and attribution text that
  do not belong in the repository.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
@karljang
karljang requested a review from a team as a code owner November 20, 2026 23:43
…onal toolchain, GPU tests

- Serve any packed sequence length: the Down and attention-output projection
  kernels are one kernel with symbolic row counts, compiled once at load; the
  sequence-length allowlist and its fallback path are removed.
- Make the CUDA toolchain optional: without nvcc, ninja or cuobjdump (or under
  python -O) the installed quantize and cast operators run, with identical
  output.
- Reuse the shared QK kernel through a rope_halves_equal option instead of a
  patched cast seam; route the shared head cast through a family hook; drop
  transformer_forward.py and kernels/rope.py.
- Bound the conditioning cache (64 keys, least recently used evicted) and pack
  GateUp weights in place.
- Ship the scenario's .cu/.cpp sources in wheels.
- Add test_fp8_kernels.py (each fused op against the unfused operators) and
  register it and test_fp8_conversions.py in l0_gr100; GPU tests skip when the
  device, toolchain or checkpoint is missing.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
@karljang
karljang requested review from a team as code owners November 21, 2026 01:42
@karljang karljang changed the title [None][fix] MiniMax-H3 static FP8 scenario: decline when it cannot run, clean docs [None][fix] MiniMax-H3 static FP8 scenario: any sequence length, optional toolchain, GPU tests Nov 21, 2026
@karljang
karljang removed request for a team, BowenFu, mzweilz, niukuo and o-stoner November 21, 2026 01:49
karljang and others added 3 commits November 20, 2026 23:32
…n path, harden load and tests

- Fix the op name in the unfused projection path (trtllm::cublas_scaled_mm);
  it raised AttributeError for inputs outside the fused kernel's contract.
- The denoise loop compiles nothing: supports() requires a kernel compiled at
  load for the shape, gate and dtypes, and checks device and layout; shape
  errors raise ValueError instead of asserting.
- Load is failure-safe: kernels compile before GateUp weights are packed, and
  an already packed block is not packed again.
- prepare() falls back to the installed operators only when a build tool is
  missing or under python -O; a failing build or qualification raises.
- Tests: unfused path for misaligned and int32 inputs, the fused branch is
  asserted, per-element bound for GateUp, built conversions through the ops,
  rope_halves_equal against the reference, non-finite values, load order.
  The kernel test no longer selects the per-process scenario settings.
- README: routing fall-through, attention backend, batch one, load behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
… tests observe the branch taken

- prepare(): a failed build or binary inspection of the optional conversion
  kernels keeps the installed operators with a warning, as a missing tool
  does; only a built kernel whose output differs from the installed operators
  fails the load.
- AdaLN launch makes the modulation indices contiguous; supports() requires
  rank-1 indices.
- Tests: the projection tests count calls into the fused kernel and the
  cuBLAS path; the unfused path is compared with the fused result (misaligned
  hidden, int32 indices, FP32 modulation); untileable shapes raise at compile;
  the built-conversion test counts calls into the built modules; the
  rope_halves_equal test uses a table with unequal halves and the FP8 launch
  shape; the load-order contract is an AST check; a stricter one-step rule for
  GateUp codes.
- README and docstrings: what compiles at load, what falls back.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>
…nce, pin the RoPE table on CPU

- The Down and attention-output projections log a warning the first time
  they take the unfused cuBLAS path, which then serves every later call for
  such inputs.
- The test that pins the family RoPE table's equal halves, which
  rope_halves_equal relies on, moves to test_minimax_h3_attention_precision.py
  (l0_cpu), so it runs on every PR.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: Kanghwan Jang <861393+karljang@users.noreply.github.com>

@trtllm-agent-review trtllm-agent-review left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. It removes the static-FP8 MiniMax-H3 scenario's sequence-length allowlist so one shared projection kernel serves every packed length, makes the load-time CUDA conversion kernels optional, and adds SM107 kernel tests.

Reviewed at commit 9bdc6f25bc (20 files, +770/−1,554). The PR moved to 249411e3e1 while the review ran; those later changes are not covered here.

Reviewed as a change to feat/minimax-h3-rubin against its merge-base (#22658).

Request changes: 2 Blockers need fixing before merge.

Blockers (2)

  • [Blocker] (B1: Correctness) tensorrt_llm/_torch/visual_gen/models/minimax_h3/minimax_h3_fp8__sm_107__u1/kernels/projection_residual.py:724: With the length allowlist gone, this kernel now serves every packed length, but the epilogue still forms the output and residual row offsets in 32 bits. row is an Int32, and mOut[(row, None)] / aux_x[(aux_row, None)] (line 729) on the compact (M, 5376) BF16 tensors become row * 5376 in 32 bits, widened only after the multiply. Compiling this file for sm_107a with the same fake tensors as compile_kernel gives mul.lo.s32 %r, row, 5376 for both. The modulation row (selected is Int64) uses mul.lo.s64, and the GateUp kernel's output row uses mul.wide.s32. From row 399,458 (399,458 x 5376 > 2^31) the address lands gigabytes before the buffer. The attention-output and Down projections then read garbage residuals and write outside out: an illegal-address fault, or silent corruption of other allocations.

    That length is reachable. validate_resolution (pipeline.py:347-366) checks only multiples of 32 and _validate_request only 5-15 s, so an explicit 2048x2048, 345-frame request packs about 415K rows; MINIMAX_H3_MAX_PIXELS caps only the auto-resolved canvas. Before this PR such lengths ran on the family cuBLAS path, and the README now promises "any packed sequence length".

    Suggested fix: form the row offset in 64 bits before slicing (e.g. row64 = cutlass.Int64(row), mOut[(row64, None)], aux_x[(cutlass.Int64(aux_row), None)]) and check that the PTX shows a wide multiply. Alternatively, reject m * n >= 2**31 in supports(). A case at 399,459 rows (about 15 GB) would pin it.

  • (N3: Ineffective new test) test_output_cast_is_exact compares hidden.float() with itself (the fixture forces the installed operators), so it cannot fail and the compiled cast/quantize branches have no CI test. (test_fp8_kernels.py:234)

Possible blockers (1)

  • (P1: Performance regression or cost) The family _head_inputs default keeps two FP32 copies of the final hidden state alive at once: about +0.82 GB per transformer call at S=38K, likely the per-rank peak under Ulysses 4. (transformer_minimax_h3.py:1459)
Nice to have (12): 9 inline, 3 below
  • (N1: Architecture and design) The FP8 scenario now drives a private NVFP4-scenario kernel through a new FP8-only flag, so an NVFP4 change can silently change the FP8 scenario's Q/K codes. (modeling.py:52)
  • (N1: Architecture and design) The optional load-time nvcc/cuobjdump conversion kernels keep a hidden second path with no stated speed gain; delete them, or justify them and make their checks fail in CI. (conversion.py:78)
  • (N1: Architecture and design) The op-level cuBLAS fallback is unreachable and calls an operator that does not exist (cublas_cublas_scaled_mm); raise on a contract violation instead. (ops.py:43)
  • (N1: Architecture and design) missing_build_tools lives in the routing-time settings module, imports cpp_extension on every route decision, and duplicates the cuobjdump lookup in conversion.py. (config.py:34)
  • (R4: Bypassing framework mechanisms) GateUp weights are packed in place from _initialize_fp8 rather than through the quant method's transform_weights hook, so the Linear's weight no longer matches its checkpoint layout. (modeling.py:317)
  • (N7: Robustness hardening) block_shared_memory_bytes reads the oversized limit, not the opt-in limit its docstring names, and raises ValueError on cuda-bindings 13.2 instead of returning 0. (projection_residual.py:791)
  • (R6: Required documentation) The settings-conflict decline covers only the FP8 route: loading an NVFP4 H3 checkpoint with scenario=auto after the FP8 scenario still fails at load. (scenarios.py:244)
  • (N6: Duplication and reuse) The new reciprocal buffers duplicate linear.inv_input_scale, which FP8QDQLinearMethod already sets to 1/input_scale at load. (modeling.py:321)
  • (N9: Validation evidence) Removing the allowlist changes the output for almost every request shape, and the PR reports no LPIPS as ENGINEERING_CRITERIA.md section 3.4 asks. (README.md:44)
  • [Nice to have] (N3: Missing or ineffective tests) - test_fp8_pipeline.py:62: assert "unary_qualified" in ... is always true, because both prepare() return paths set the key. Assert q["unary_qualified"] or q["reason"].startswith("not found:") instead.
    • test_fp8_kernels.py: every input is fresh and aligned, so nothing reaches the fallback branches of ops.down/ops.attention_projection. That is how the cublas_cublas_scaled_mm call got through. The length tests also never check that projection_residual.run ran. Add a 64-byte-offset or non-contiguous hidden case that asserts not supports(...), and count run calls in the length tests.
    • modeling.py:52: no test runs rope_halves_equal=True. Parametrize test_h3_scenario_fused_fp8_qk.py over it; its RoPE inputs already have equal halves.
    • setup.py:250: the new precompiled-extraction rule has no test. Add a staticQuant.cu case to test_precompiled_preserves_source_schemas.
    • test_fp8_contracts.py: the family-path stubs in _fused_dispatch_namespace (lines 722-864) only served the deleted fallback test. Line 314 simulates a "need nvcc" error that select() can no longer raise; use apply's real conflict message instead.
    • test_fp8_kernels.py:54: the ops under test do not need config.select(). When the whole minimax_h3 test directory runs in one process, it skips all 38 cases (CI is unaffected, since it runs one process per entry).
    • test_fp8_kernels.py:165/210: bit-identity with cuBLASLt from 1000 rows depends on cuBLAS's algorithm choice. The GateUp comment at 182-183 is wrong (the kernel rounds the GEMM output to BF16 too), and its 0.97 ratio allows twice the expected mismatch.
  • [Nice to have] (R3: Repository rules) - test_fp8_conversions.py:22 skips whenever prepare() returns no kernels. prepare() also returns none after a failed build or a failed byte-identity check, so this test, newly registered on GR100, reports SKIPPED on exactly the regression it exists for. tests/unittest/_torch/visual_gen/AGENTS.md: "unbuilt or unimportable first-party modules ... must fail loudly, not pytest.skip ... keep the gate as a skip and fail only on the resource half". Skip only when missing_build_tools() is non-empty, and fail with proof["reason"] otherwise.
    • test_fp8_pipeline.py:31 turns a missing MiniMax-H3-FP8 checkpoint into a skip. The same file forbids that ("This covers missing model checkpoints"); call get_checkpoint directly, as before.
    • projection_residual.py:790, 798, 851, 874 and ops.py:107 are unannotated. CODING_GUIDELINES.md: "Always annotate functions."
    • conversion.py:207 catches RuntimeError (which includes CUDA faults and OOM) and bare-assert failures around the whole build and qualification. A failed check is logged only as "AssertionError:" at warning level, which the default level hides, and a sticky CUDA fault resurfaces later at torch.reciprocal(scales). CODING_GUIDELINES.md: "limit the except to the smallest set of errors possible." Catch only around build() and the cuobjdump call, and raise explicit qualification errors.
  • [Nice to have] (N5: Code tidiness) - modeling.py:114-121 and 74-86: with the length fallback gone, Fp8Attention's BF16 branch and Fp8QkvMethod's BF16 re-quantize and super().apply fallback are unreachable. The block always passes FP8 codes, and every guard is validated in _initialize_fp8. The docstrings at lines 71 and 104 still describe the removed paths.
    • projection_residual.py:639-666 keeps the GateUp epilogue comments ("... SG -> E4M3 -> global", "gate 0..3 | up 4..7").
    • conditioning.py:125: eviction scans every entry for entry[1] == old, so it depends on the key tuple built at line 58, and evictions is read only by the test. Nesting the tables under the request key makes eviction a single pop.
What this review checked
  • The merged projection kernel keeps the attention-output path as it ran: the old attention_output.py differed from down_residual.py only in the gate column, and #22658 overrode its config to the same stages and swizzle the shared CONFIG uses.
  • rope_halves_equal=True reproduces the deleted half_rope_tensor_cast exactly and is valid for every request shape, because every H3 RoPE table is cat((freqs, freqs)) and padding uses cos=1/sin=0; the NVFP4 default path is unchanged.
  • The family forward plus the new _head_inputs hook matches the deleted shared_output_forward line for line apart from the head inputs, and the projection epilogue guards every non-TMA access past M with row < m_total.
  • Not covered: No GPU run: no SM107 (212-SM) device was available; the 32-bit offset finding rests on PTX compiled for sm_107a from this kernel.
  • Not covered: Overlap with other open PRs on feat/minimax-h3-rubin was not checked.
  • Not covered: The CuTe kernels' TMEM and CLC pipelines were reviewed only around the changed epilogue and host code.

Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.

assert len(down_kernels) == 1


def test_output_cast_is_exact(scenario):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Blocker] (N3: Ineffective new test) This test cannot fail. The module fixture sets conversion._MODULES[device] = (None, None) (lines 66-68). So ops.output_cast takes the cast is None branch and returns hidden.float(), and the assertion compares hidden.float() with itself. For the same reason, no test in this file runs the compiled cast in ops.output_cast or the compiled quant.quantize_config(...) branch of ops.attention_projection (ops.py:154-164). Those branches are the production path whenever nvcc, ninja and cuobjdump are present, and this PR opens them to every sequence length. test_fp8_conversions.py calls the extensions directly, not through the ops, and skips when they fail.

Suggested fix: parametrize the scenario fixture over the installed and the compiled conversions (build with prepare, skip only when missing_build_tools() is non-empty). Keep the references on the installed operators, and drop this test from the installed variant. The compiled variant then checks ops.output_cast and the quant_factor wiring at every length.

f"{preview}{suffix}"
)

def _head_inputs(self, hidden_states: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Possible blocker] (P1: Performance regression or cost) This default makes both FP32 casts up front, and video_input/audio_input stay referenced through both projections (lines 1358-1360). So the family BF16 pipeline and the NVFP4 scenario now hold two [B, S, 5376] FP32 copies of the final hidden state at once; before, each cast lived only inside its projection call. Only the transformer blocks are compiled, so this runs eagerly: about +0.82 GB of peak memory per transformer call at S38K and +2.2 GB at S104K. Under Ulysses 4 the heads run on the gathered full sequence, so this is likely the per-rank peak.

Could you compare max_memory_allocated() around one forward (768x1344x124 and a 15 s clip, family at Ulysses 1, NVFP4 at Ulysses 4) before and after? The fix is cheap either way. Cast once and return the same tensor twice when both heads take the same dtype (proj_out.has_any_quant == audio_proj_out.has_any_quant), or release the first input before the second cast. The values stay bit-identical.

valid_len,
0,
cast_g_kernel=half_rope_tensor_cast,
rope_halves_equal=True,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) The FP8 scenario now drives the NVFP4 scenario's private _qk_norm_rope_fp8_impl and adds an FP8-only rope_halves_equal mode to that scenario's kernel file. The scenario layout (scenarios.py:17-19) says each folder holds a complete variant "with its own kernels ... the shared code untouched". No CI test runs HALF_EQ=True, so an NVFP4 edit to _group_vectors or _tensor_cast_g_kernel can silently change FP8 Q/K. Any of these would fix it:

  • move fused_fp8_qk.py to the family next to fused_adaln.py, with a public entry point and its modes passed as arguments;
  • since every H3 RoPE table is cat((freqs, freqs)) (transformer_minimax_h3.py:152), make the half reuse unconditional and call the public op;
  • keep the FP8 copy.

Whichever you pick, add a test for the mode this scenario runs.

verbose=True,
)
try:
return load(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) The scalar quantize and output-cast kernels are still built with nvcc/ninja at every model load and checked against cuobjdump SASS regexes. Now they are also optional: every failure falls back behind a warning that the default error log level hides. The PR calls the installed operators byte-identical and gives no speed number for the built kernels. So which path a deployment runs depends on PATH and -O, and a toolkit update that changes the SASS quietly turns them off.

Unless a measured gain justifies them, consider deleting them (with pdl.py, the .cu/.cpp sources and the setup.py special case) and always using static_quantize_e4m3_per_tensor / .float(). If they stay, state the gain, build them ahead of time (or in Triton like adaln.py), and turn the byte and SASS checks into CI tests that fail.

weight_scale: torch.Tensor,
) -> torch.Tensor:
return torch.ops.trtllm.cublas_scaled_mm(
return torch.ops.trtllm.cublas_cublas_scaled_mm(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N1: Architecture and design) This calls torch.ops.trtllm.cublas_cublas_scaled_mm, which does not exist. Only trtllm::cublas_scaled_mm is registered (cpp/tensorrt_llm/thop/cublasScaledMM.cpp:366-376); the _scaled_mm -> _cublas_scaled_mm rename also hit the op name. So the fallback in down and attention_projection raises AttributeError whenever projection_residual.supports() is False.

No caller reaches that branch today: the block passes fresh, contiguous, aligned tensors and already raises for any other layout (modeling.py:171-177). I'd make that strict here too: raise a ValueError naming the failed condition and delete _cublas_scaled_mm and _gated_residual, so each op keeps one qualified numerics path. If the fallback stays, restore cublas_scaled_mm and add a test that forces it (e.g. a 64-byte-offset hidden).

block.register_buffer("_fp8_packed_gate_up", packed, persistent=False)
# The kernel's tile order replaces the checkpoint's row order in place of a second copy.
gate_up = block.ff.gate_up_proj
gate_up.weight.data = gate_up_swiglu.pack_weight(gate_up.weight)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R4: Bypassing framework mechanisms) The packed layout is applied here by swapping weight.data and installing a method whose apply raises. The framework hook for this is the quant method's transform_weights (linear.py:569, called from post_load_weights) with replace_parameter_and_save_metadata (_torch/utils.py:718); FP8BlockScalesLinearMethod.transform_weights (linear.py:1692) uses it for an SM107 layout. As it stands, the GateUp Linear keeps its checkpoint name, shape and has_fp8_qdq while holding rows in tile order. state_dict() exports permuted rows, and any reload path would write checkpoint-order rows that never get repacked, because initialize_fp8 returns early. Installing PackedGateUpMethod when the block is built and packing in its transform_weights keeps the in-place memory saving and lets the method own the layout.

str(idx.dtype),
repr(sorted(_CFG.items())),
def block_shared_memory_bytes(device_index):
"""Shared memory one block of this device may opt in to, 0 when the driver does not report it."""

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N7: Robustness hardening) The docstring says "may opt in to, 0 when the driver does not report it", but attribute 150 is the oversized per-block limit, not the opt-in limit that gate_up_swiglu._smem_optin_bytes reads. And cuda.CUdevice_attribute(150) raises ValueError: 150 is not a valid CUdevice_attribute on cuda-bindings 13.2, which cuda-python>=13 allows, before the driver is asked. On such an install the FP8 load fails with that enum error instead of the capacity message. Please name the attribute once (a constant with the CUDA enum name), return 0 on the ValueError as documented, and name the helper after the limit it reads. The GateUp check in modeling.py:305-308 could then move into gate_up_swiglu._get_compiled, as compile_kernel does for its own kernel.

try:
select()
except RuntimeError as error:
raise NoScenario(f"the static-FP8 fusion cannot run in this process: {error}") from error

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (R6: Required documentation) This decline covers one order only. route_nvfp4_sm_107_u1_u4 still calls pkg.config.apply(...) unguarded (line 154), and resolve_scenario catches only NoScenario. So a process that already loaded the static-FP8 scenario fails at load with "this process already runs the scenario with other settings" when it then loads an NVFP4 H3 checkpoint with visual_gen_scenario="auto" (the presets differ in six fields). The sleep/wake flow ("switch between large checkpoints on one GPU") is exactly that case. Translating the conflict once covers both routes: for example, a dedicated exception from config.apply that resolve_scenario turns into a decline under auto and an error with the reason under require. If not, the README should say the decline holds in this order only. A resolver test with a pre-selected conflicting config would pin it.

inverse = torch.empty_like(linear.input_scale)
_reciprocal[(1,)](linear.input_scale, inverse, num_warps=4)
# FP32 division on the host is correctly rounded, as the kernels' div.rn is.
inverse = torch.reciprocal(linear.input_scale.detach().cpu()).to(device)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N6: Duplication and reuse) FP8QDQLinearMethod already keeps exactly this value in linear.inv_input_scale: 1.0 / input_scale at load (linear.py:1027), and 1.0 / max_input_scale for the fused QKV/GateUp loaders (linear.py:1116). Both are correctly rounded FP32, so the bits match. Registering or reading linear.inv_input_scale avoids a second copy and the 100 device-host round trips at load.

- Ulysses, TP and parallel VAE are one. GB300 and four-GPU inference keep their
separately qualified family routes.
shapes. BF16 or mixed checkpoints are not admitted.
- Batch one, any packed sequence length. The fused kernels take the row count at

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] (N9: Validation evidence) This changes the output for nearly every request. Before, only packed lengths 37776, 37820 and 38169 ran the fused kernels; every other length, including the 37712-row loader warmup, ran the family math. The PR table says only "output differs from the fallback, as expected", while ENGINEERING_CRITERIA.md section 3.4 asks for LPIPS against the golden for lossless fusion work, and the README drops its note that this integration needs its own LPIPS qualification. Could you add LPIPS (and the audio metric test_minimax_h3_e2e.py uses) for one previously-fallback shape, measured against #22658's output?

@karljang
karljang merged commit 8868c71 into NVIDIA:feat/minimax-h3-rubin Nov 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants