Skip to content

[https://nvbugs/7003734][test] Pin LTX-2 VisualGen worker NVFP4 GEMMs to CUTLASS and refresh golden - #22724

Merged
chang-l merged 3 commits into
NVIDIA:mainfrom
chang-l:nvbug/7003734-ltx2-pin-cutlass
Nov 21, 2026
Merged

chang-l merged 3 commits into
NVIDIA:mainfrom
chang-l:nvbug/7003734-ltx2-pin-cutlass

Conversation

@chang-l

@chang-l chang-l commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

  • The change is test-only. It does not modify library source.
  • The new _fixed_nvfp4_gemm_backends() helper pins nvfp4_gemm_allowed_backends to ["cutlass"]. The in-process golden helper and the spawned-worker wrapper both use it.
  • The spawned-worker wrapper wraps executor.run_diffusion_worker. It builds each worker's pipeline inside the pin context. The wrapper rejects calls in the process where the executor function is patched.
  • The cache validator requires each trtllm::nvfp4_gemm::gemm entry to be keyed on ('cutlass',). It accepts tactic -1 or a tactic whose backend is allowed. A failure means the pin did not apply.
  • The FP4 activation quantizer stays autotuned. The commit states its TRT-LLM and FlashInfer tactics are byte-identical.
  • The golden video and its JSON sidecar were regenerated outside CI, using a release image and a one-file overlay. The author states the exact CI GEMM assignment could not be reconstructed.
  • The golden JSON provenance removes the prior CI job, build, and artifact fields. Traceability to the CI run that produced the old golden is reduced.
  • The golden ZIP (visual_gen_lpips_golden_media_ltx2.zip) changes. The author reports its other four members are byte-identical. Its contents were not inspected.

QA Engineer Review

  • Modified: tests/integration/defs/examples/visual_gen/test_visual_gen_ltx2.py (test_ltx2_cuda_graph_trtllm_backend). Adds a worker-runner patch and a post-run autotuner cache assertion.
  • Modified: tests/integration/defs/examples/visual_gen/visual_gen_test_utils.py. Adds the fixed-backend worker wrapper, cache validator, and shared backend helper.
  • Modified: tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_trtllm_backend_lpips_golden_video.json. Refreshed golden metadata and provenance.
  • Modified: tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/visual_gen_lpips_golden_media_ltx2.zip. Refreshed golden media archive.
  • No test-list files changed. Test-db or QA list membership: not available from the supplied diff.
  • Thresholds stay at 0.15 (against golden) and 0.01 (eager versus graph).
  • Author-reported manual B200 runs: the cache assertion passed in every run, with 42 GEMM decisions pinned to CUTLASS. Thirteen runs across three nodes and eight containers gave identical reference and graph hashes. These are not CI results.
  • The /bot run requests did not select this B200-only test. Whether it ran in CI: not available.
  • Coverage verdict: needs follow-up. The first post-merge B200 CI runs containing this PR must confirm the pin and the thresholds.

Per-File QA Perspective

  • test_visual_gen_ltx2.py: Covers the LTX-2 CUDA-graph path with the CUTLASS pin. Not in test-db or qa lists in the supplied diff; verify the B200 post-merge CI listing.
  • visual_gen_test_utils.py: Adds the worker wrapper and cache validator. Verify the validator fails when the pin is absent.
  • Golden JSON: Changes reference metadata and provenance. Verify LPIPS still passes in CI.
  • Golden ZIP: Replaces reference media. Verify it matches the refreshed JSON in CI.

Description

Follow-up to #21220 for https://nvbugs/7003734.

Problem. test_ltx2_cuda_graph_trtllm_backend (from #19080) compares the VisualGen worker's NVFP4 LTX-2 video against a fixed golden, with a limit of LPIPS < 0.15.

  • The spawned worker autotunes each NVFP4 GEMM shape by timing across cutlass, cublaslt, cutedsl and cuda_core.
  • Those backends are not bit-identical, so the pick, and therefore the output, can change between machines and runs.
  • The 8-step two-stage LTX-2 pipeline amplifies any bit difference. Every non-identical output we measured scored 0.14–0.24 against the golden: 14 of 17 variants failed, 3 passed by chance. Flipping a single GEMM shape from cuBLASLt to CUTLASS was enough to score 0.197–0.220. In practice the check is a bit-exactness check.
  • Post-merge B200 CI failed with two distinct outputs. One scored 0.190425 in 7 builds, the other 0.189465 in 2. Our B200 nodes reproduce the golden exactly.

The NVFP4 activation-quantizer bug fixed in #21220 is real, but it is not what CI hit:

  • Forcing it on any subset of the affected shapes gives 0.145–0.206. None of those outputs matches either CI video.
  • That buggy tactic is 18–30x slower, so it never wins the autotuner race (0 of 246 timings).

Why not raise the threshold. That same quantizer bug scores 0.193 on this test when forced. Any threshold wide enough to absorb the 0.14–0.24 backend spread would also pass it.

Fix (test only, no product code or public API change).

  • Pin the spawned workers' NVFP4 GEMMs to CUTLASS. This is the GEMM half of the policy that _fixed_nvfp4_quantization_backend already applies to the in-process NVFP4 goldens.
    • That helper sets a thread-local in the calling process, so it cannot reach a worker that VisualGen spawns.
    • Instead, the generator patches the worker entry point executor.run_diffusion_worker while it constructs VisualGen (the executor unit tests already patch this attribute).
    • Each spawned worker then builds its pipeline inside model_extra_attrs({"nvfp4_gemm_allowed_backends": ["cutlass"]}).
    • Autotuning (now among CUTLASS tiles), torch.compile and CUDA graphs are unchanged.
  • Prove the pin took effect. After both workers run, the test reads their shared autotuner cache. Every trtllm::nvfp4_gemm::gemm decision must be keyed on ('cutlass',) and use a CUTLASS tactic. If the pin ever stops reaching a worker, the test fails with that message rather than an LPIPS mismatch.
  • Leave the other tuned ops tuned. The FP4 activation quantizer and the fused CuTeDSL GEMM+GELU stay autotuned, because their tactic choices give byte-identical output (perturbation rows below). The test therefore still covers the FlashInfer quantizer path fixed in [https://nvbugs/7003734][fix] Preserve NVFP4 activation row and scale layouts #21220.
  • Regenerate the golden. The pinned output scores 0.190445 against the old golden, so the old one cannot be kept.
    • The new ltx2_trtllm_backend_lpips_golden_video.mp4 is the pinned graph-off worker video.
    • The other four members of visual_gen_lpips_golden_media_ltx2.zip are byte-identical.
    • The sidecar records "nvfp4_gemm_backend": "cutlass" (as the FLUX NVFP4 goldens do), plus image, digest, driver, overlay and job provenance.
  • Thresholds are unchanged: 0.15 against the golden, 0.01 for eager vs. graph.

Alternatives considered.

  • enable_autotune=False would drop the worker-autotune and tuned-graph-capture path that this test was restored to cover.
  • A VisualGen NVFP4 GEMM backend option (like the LLM API's nvfp4_gemm_config) would be a public-API change and belongs in its own PR.

Test Coverage

examples/visual_gen/test_visual_gen_ltx2.py::test_ltx2_cuda_graph_trtllm_backend runs only in B200 post-merge (DGX_B200-PyTorch-Post-Merge-1/2), so a plain /bot run does not select it. It was validated with the real pytest on B200 outside CI:

Check Result
Final code of this PR vs. refreshed golden LPIPS 0.000000, on two nodes
Refreshed golden on a node other than its producer LPIPS 0.000000
Graph vs. eager (pinned) LPIPS 0.000000
Determinism 13 pinned runs on 3 nodes and 8 containers: reference = graph = sha256 00d73585… every time
Cache assertion Passed in every run; 42 NVFP4 GEMM decisions, all ('cutlass',)
Natural CUTLASS tile noise Nodes picked different CUTLASS tiles for 18–29 of the 42 keys; bytes identical
CUTLASS tiles forced (3 seeds incl. slowest; all 42 keys moved, 59/59 seeds honored) Byte-identical
Fused CuTeDSL GEMM+GELU tiles forced (3 seeds; all 4 keys moved) Byte-identical
FP4 quantizer forced: FlashInfer-only vs. TRT-LLM-only (13/13 shapes each) Byte-identical
Unpinned (old) test vs. refreshed golden (control) 0.190445, fails, as expected

Residual risks.

  • CI's exact GEMM assignment could not be reconstructed, because CI does not keep the autotuner cache. The first post-merge B200 runs that include this PR are the real confirmation.
  • Validation ran on ComputeLab B200 with a release image plus a one-file overlay, not on the CI cluster or a CI-built wheel.
  • As before, any legitimate numerical change to this pipeline (kernels, compile, VAE) will need a golden refresh.

Links

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

… to CUTLASS and refresh golden

test_ltx2_cuda_graph_trtllm_backend compares the VisualGen worker's NVFP4
LTX-2 video with a fixed golden at LPIPS < 0.15. The worker autotunes each
NVFP4 GEMM shape by timing across cutlass, cublaslt, cutedsl and cuda_core.
These backends are not bit-identical, so the pick, and with it the output,
can change between machines, and the 8-step two-stage pipeline turns any bit
difference into a golden LPIPS of 0.14-0.24. Post-merge B200 CI failed with
two such outputs (0.190425, 0.189465) while our B200 nodes reproduced the
golden exactly.

Pin the spawned workers' NVFP4 GEMMs to CUTLASS, the GEMM policy that
_fixed_nvfp4_quantization_backend already applies to the in-process NVFP4
goldens. That helper sets a thread-local in the calling process and cannot
reach a worker VisualGen spawns, so the generator patches the worker entry
point while constructing VisualGen and each worker builds its pipeline under
the pin. The test checks the shared autotuner cache to prove both workers
were pinned. The FP4 activation quantizer and the fused CuTeDSL GEMM+GELU
stay autotuned: their tactic choices give byte-identical output.

Regenerate the ltx2_trtllm_backend golden from the pinned path (the other
four ltx2 zip members are unchanged) and record the pin and its provenance
in the sidecar. Thresholds are unchanged.

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l
chang-l requested a review from a team as a code owner November 21, 2026 00:41
@chang-l

chang-l commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86479 [ run ] triggered by Bot. Commit: 6671a7e Link to invocation

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The LTX-2 visual-generation test pins NVFP4 GEMM tuning to CUTLASS in the diffusion worker and validates the shared autotuner cache after video generation. The golden-video metadata and provenance are updated.

Changes

Visual-generation backend pinning

Layer / File(s) Summary
Backend pinning and cache validation
tests/integration/defs/examples/visual_gen/visual_gen_test_utils.py
Shared constants define the allowed NVFP4 GEMM backend. A worker wrapper applies the setting, and a cache validator checks matching entries and tactics.
Integration test and golden metadata
tests/integration/defs/examples/visual_gen/test_visual_gen_ltx2.py, tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_trtllm_backend_lpips_golden_video.json
The LTX-2 test uses the wrapper during VisualGen construction and validates the cache after video generation. The golden metadata and provenance are updated.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Sequence Diagram(s)

sequenceDiagram
  participant VisualGenTest
  participant Executor
  participant SpawnedWorker
  participant AutotunerCache
  VisualGenTest->>Executor: Temporarily replace diffusion-worker runner
  VisualGenTest->>Executor: Construct VisualGen and generate videos
  Executor->>SpawnedWorker: Run fixed-backend worker wrapper
  SpawnedWorker->>Executor: Apply backend pin and call run_diffusion_worker
  VisualGenTest->>AutotunerCache: Validate NVFP4 GEMM entries
Loading
Merge Risk: Minimal · up to 7554d

This change pins the LTX-2 test's NVFP4 GEMMs to CUTLASS and refreshes the golden video so the test is deterministic. It affects only test code and golden metadata, with no production impact. The first post-merge B200 runs will confirm it, and no merge-blocking risk remains.

Security architecture risk: Minimal · up to 7554d

The change constrains numerical backend selection for an integration test and checks its recorded tuning decisions. No introduced or worsened risk materially affecting security, failure containment, rollback, or ownership was identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: pinning LTX-2 VisualGen worker NVFP4 GEMMs to CUTLASS and refreshing the golden. The changeset adds the worker-side GEMM pin, the cache assertion, and the regenerated …
Description check Passed The description covers the problem, the fix, the alternatives considered, test coverage, residual risks, and links. It follows the template's Description, Test Coverage, and PR Checklist sections, and…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: pinning LTX-2 VisualGen worker NVFP4 GEMMs to CUTLASS and refreshing the golden. The changeset adds the worker-side GEMM pin, the cache assertion, and the regenerated golden video and metadata, so the title matches the primary change. It follows the bracketed NVBugs ID format. It lacks a [type] tag (for example [test]). The title includes [test], so the format is satisfied.

Full details: Description check

Explanation

The description covers the problem, the fix, the alternatives considered, test coverage, residual risks, and links. It follows the template's Description, Test Coverage, and PR Checklist sections, and it explains the what and why in detail. The template's section for the title format is not applicable to the body. The checklist is marked as reviewed. The description is thorough and mostly complete.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7f2f124 and 7554da9.

Files selected for processing: 1

  • tests/integration/defs/examples/visual_gen/visual_gen_test_utils.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@trtllm-agent-review trtllm-agent-review left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. It pins the spawned VisualGen worker's NVFP4 GEMMs to CUTLASS in test_ltx2_cuda_graph_trtllm_backend, checks from the shared autotuner cache that every GEMM was tuned under the pin, and refreshes the golden video to the pinned output.

Reviewed at commit 6671a7e319 (4 files, +97/−15).

Test-only change: the review covered the four changed files and the VisualGen executor, autotuner and NVFP4 GEMM code they rely on.

Checked and set aside: leaving the FP4 activation quantizer autotuned (its two tactics are bit-identical), the golden's origin in a release image plus one overlaid file (nothing else on this test's numerics path changed between the image's commit and the base), the external-launch path (unreachable with one worker) and the cache-key string match (it fails, not passes, if the key format changes).

Approve: no Blocker found. 2 suggestions below are nice to have.

Nice to have (2): 2 inline, 0 below
  • (N6: Duplication and reuse) The worker pin copies the GEMM half of _fixed_nvfp4_quantization_backend and leaves the FP4 quantizer tuned; only the PR description says that is on purpose. (visual_gen_test_utils.py:664)
  • (N8: Documentation improvements) The golden's provenance records an internal run id, Slurm job id and node name that no outside reader can resolve; record the pytest node instead. (ltx2_trtllm_backend_lpips_golden_video.json:38)
What this review checked
  • Every NVFP4 GEMM in the worker gets the pin: the worker builds its pipeline in its main thread inside run_diffusion_worker (executor.py:880-889), Linear reads nvfp4_gemm_allowed_backends from the thread-local extra attrs at construction (linear.py:4503-4512), and the spawn target is read as the module global at executor.py:1050 before the patch is undone.
  • _assert_nvfp4_gemm_cache_pinned can fail: the cache key carries str(runner.unique_id()) with the allowed-backends tuple (autotuner.py:676-691, torch_custom_ops.py:1434-1438), and on caches in the AutoTuner's save format it passes pinned ones and fails unpinned, mixed and empty ones.
  • The refreshed zip changes only ltx2_trtllm_backend_lpips_golden_video.mp4: its sha256 matches the sidecar, the other four members are byte-identical, and the recorded overlay blob equals the base's torch_custom_ops.py.
  • Not covered: no GPU run, and the B200 post-merge stages that run this test have not run on this commit.
  • Not covered: the refreshed golden video was not decoded or viewed.
  • Not covered: no ruff or pyrefly run on the changed files.

Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.

@github-actions

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86479 [ run ] completed with state SUCCESS. Commit: 6671a7e
/LLM/main/L0_MergeRequest_PR pipeline #71669 completed with status: 'SUCCESS'

CI Report

Link to invocation

chang-l and others added 2 commits November 20, 2026 20:25
…pips/ltx2_trtllm_backend_lpips_golden_video.json

Co-authored-by: trtllm-agent-review <svctrtllm-review@nvidia.com>
Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
…olden helpers

Move the GEMM half of _fixed_nvfp4_quantization_backend into
_fixed_nvfp4_gemm_backends() and use it from both the in-process helper
(which keeps its FlashInfer flip) and the spawned-worker wrapper. Document
why the worker wrapper leaves the FP4 activation quantizer autotuned: its
TRT-LLM and FlashInfer tactics are bit-identical
(test_fp4_quantize_flashinfer.py). No behavior change.

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l

chang-l commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86513 [ run ] triggered by Bot. Commit: 7554da9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86513 [ run ] completed with state SUCCESS. Commit: 7554da9
/LLM/main/L0_MergeRequest_PR pipeline #71700 completed with status: 'SUCCESS'

CI Report

Link to invocation

@chang-l
chang-l merged commit 260e6b8 into NVIDIA:main Nov 21, 2026
15 checks passed
@github-actions

Copy link
Copy Markdown

✅ LFS objects already in storage (1 file) — no sync needed.

These LFS-tracked files are already present in this repository's LFS storage:

  • tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/visual_gen_lpips_golden_media_ltx2.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants