Repository navigation
[https://nvbugs/7003734][test] Pin LTX-2 VisualGen worker NVFP4 GEMMs to CUTLASS and refresh golden - #22724
Conversation
… to CUTLASS and refresh golden test_ltx2_cuda_graph_trtllm_backend compares the VisualGen worker's NVFP4 LTX-2 video with a fixed golden at LPIPS < 0.15. The worker autotunes each NVFP4 GEMM shape by timing across cutlass, cublaslt, cutedsl and cuda_core. These backends are not bit-identical, so the pick, and with it the output, can change between machines, and the 8-step two-stage pipeline turns any bit difference into a golden LPIPS of 0.14-0.24. Post-merge B200 CI failed with two such outputs (0.190425, 0.189465) while our B200 nodes reproduced the golden exactly. Pin the spawned workers' NVFP4 GEMMs to CUTLASS, the GEMM policy that _fixed_nvfp4_quantization_backend already applies to the in-process NVFP4 goldens. That helper sets a thread-local in the calling process and cannot reach a worker VisualGen spawns, so the generator patches the worker entry point while constructing VisualGen and each worker builds its pipeline under the pin. The test checks the shared autotuner cache to prove both workers were pinned. The FP4 activation quantizer and the fused CuTeDSL GEMM+GELU stay autotuned: their tactic choices give byte-identical output. Regenerate the ltx2_trtllm_backend golden from the pinned path (the other four ltx2 zip members are unchanged) and record the pin and its provenance in the sidecar. Thresholds are unchanged. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
|
/bot run |
|
PR_Github #86479 [ run ] triggered by Bot. Commit: |
|
No actionable comments were generated in the recent review. WalkthroughThe LTX-2 visual-generation test pins NVFP4 GEMM tuning to CUTLASS in the diffusion worker and validates the shared autotuner cache after video generation. The golden-video metadata and provenance are updated. ChangesVisual-generation backend pinning
Estimated code review effort: 2 (Simple) | ~10 minutes Change: Other Sequence Diagram(s)sequenceDiagram
participant VisualGenTest
participant Executor
participant SpawnedWorker
participant AutotunerCache
VisualGenTest->>Executor: Temporarily replace diffusion-worker runner
VisualGenTest->>Executor: Construct VisualGen and generate videos
Executor->>SpawnedWorker: Run fixed-backend worker wrapper
SpawnedWorker->>Executor: Apply backend pin and call run_diffusion_worker
VisualGenTest->>AutotunerCache: Validate NVFP4 GEMM entries
Merge Risk:
|
There was a problem hiding this comment.
Thanks for the PR. It pins the spawned VisualGen worker's NVFP4 GEMMs to CUTLASS in test_ltx2_cuda_graph_trtllm_backend, checks from the shared autotuner cache that every GEMM was tuned under the pin, and refreshes the golden video to the pinned output.
Reviewed at commit 6671a7e319 (4 files, +97/−15).
Test-only change: the review covered the four changed files and the VisualGen executor, autotuner and NVFP4 GEMM code they rely on.
Checked and set aside: leaving the FP4 activation quantizer autotuned (its two tactics are bit-identical), the golden's origin in a release image plus one overlaid file (nothing else on this test's numerics path changed between the image's commit and the base), the external-launch path (unreachable with one worker) and the cache-key string match (it fails, not passes, if the key format changes).
Approve: no Blocker found. 2 suggestions below are nice to have.
Nice to have (2): 2 inline, 0 below
- (N6: Duplication and reuse) The worker pin copies the GEMM half of _fixed_nvfp4_quantization_backend and leaves the FP4 quantizer tuned; only the PR description says that is on purpose. (
visual_gen_test_utils.py:664) - (N8: Documentation improvements) The golden's provenance records an internal run id, Slurm job id and node name that no outside reader can resolve; record the pytest node instead. (
ltx2_trtllm_backend_lpips_golden_video.json:38)
What this review checked
- Every NVFP4 GEMM in the worker gets the pin: the worker builds its pipeline in its main thread inside run_diffusion_worker (executor.py:880-889), Linear reads nvfp4_gemm_allowed_backends from the thread-local extra attrs at construction (linear.py:4503-4512), and the spawn target is read as the module global at executor.py:1050 before the patch is undone.
- _assert_nvfp4_gemm_cache_pinned can fail: the cache key carries str(runner.unique_id()) with the allowed-backends tuple (autotuner.py:676-691, torch_custom_ops.py:1434-1438), and on caches in the AutoTuner's save format it passes pinned ones and fails unpinned, mixed and empty ones.
- The refreshed zip changes only ltx2_trtllm_backend_lpips_golden_video.mp4: its sha256 matches the sidecar, the other four members are byte-identical, and the recorded overlay blob equals the base's torch_custom_ops.py.
- Not covered: no GPU run, and the B200 post-merge stages that run this test have not run on this commit.
- Not covered: the refreshed golden video was not decoded or viewed.
- Not covered: no ruff or pyrefly run on the changed files.
Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.
|
Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements. |
|
PR_Github #86479 [ run ] completed with state |
…pips/ltx2_trtllm_backend_lpips_golden_video.json Co-authored-by: trtllm-agent-review <svctrtllm-review@nvidia.com> Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
…olden helpers Move the GEMM half of _fixed_nvfp4_quantization_backend into _fixed_nvfp4_gemm_backends() and use it from both the in-process helper (which keeps its FlashInfer flip) and the spawned-worker wrapper. Document why the worker wrapper leaves the FP4 activation quantizer autotuned: its TRT-LLM and FlashInfer tactics are bit-identical (test_fp4_quantize_flashinfer.py). No behavior change. Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
|
/bot run |
|
PR_Github #86513 [ run ] triggered by Bot. Commit: |
|
PR_Github #86513 [ run ] completed with state |
|
✅ LFS objects already in storage (1 file) — no sync needed. These LFS-tracked files are already present in this repository's LFS storage:
|
Dev Engineer Review
_fixed_nvfp4_gemm_backends()helper pinsnvfp4_gemm_allowed_backendsto["cutlass"]. The in-process golden helper and the spawned-worker wrapper both use it.executor.run_diffusion_worker. It builds each worker's pipeline inside the pin context. The wrapper rejects calls in the process where the executor function is patched.trtllm::nvfp4_gemm::gemmentry to be keyed on('cutlass',). It accepts tactic-1or a tactic whose backend is allowed. A failure means the pin did not apply.visual_gen_lpips_golden_media_ltx2.zip) changes. The author reports its other four members are byte-identical. Its contents were not inspected.QA Engineer Review
tests/integration/defs/examples/visual_gen/test_visual_gen_ltx2.py(test_ltx2_cuda_graph_trtllm_backend). Adds a worker-runner patch and a post-run autotuner cache assertion.tests/integration/defs/examples/visual_gen/visual_gen_test_utils.py. Adds the fixed-backend worker wrapper, cache validator, and shared backend helper.tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_trtllm_backend_lpips_golden_video.json. Refreshed golden metadata and provenance.tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/visual_gen_lpips_golden_media_ltx2.zip. Refreshed golden media archive./bot runrequests did not select this B200-only test. Whether it ran in CI: not available.Per-File QA Perspective
test_visual_gen_ltx2.py: Covers the LTX-2 CUDA-graph path with the CUTLASS pin. Not in test-db or qa lists in the supplied diff; verify the B200 post-merge CI listing.visual_gen_test_utils.py: Adds the worker wrapper and cache validator. Verify the validator fails when the pin is absent.Description
Follow-up to #21220 for https://nvbugs/7003734.
Problem.
test_ltx2_cuda_graph_trtllm_backend(from #19080) compares the VisualGen worker's NVFP4 LTX-2 video against a fixed golden, with a limit of LPIPS < 0.15.cutlass,cublaslt,cutedslandcuda_core.The NVFP4 activation-quantizer bug fixed in #21220 is real, but it is not what CI hit:
Why not raise the threshold. That same quantizer bug scores 0.193 on this test when forced. Any threshold wide enough to absorb the 0.14–0.24 backend spread would also pass it.
Fix (test only, no product code or public API change).
_fixed_nvfp4_quantization_backendalready applies to the in-process NVFP4 goldens.executor.run_diffusion_workerwhile it constructsVisualGen(the executor unit tests already patch this attribute).model_extra_attrs({"nvfp4_gemm_allowed_backends": ["cutlass"]}).trtllm::nvfp4_gemm::gemmdecision must be keyed on('cutlass',)and use a CUTLASS tactic. If the pin ever stops reaching a worker, the test fails with that message rather than an LPIPS mismatch.ltx2_trtllm_backend_lpips_golden_video.mp4is the pinned graph-off worker video.visual_gen_lpips_golden_media_ltx2.zipare byte-identical."nvfp4_gemm_backend": "cutlass"(as the FLUX NVFP4 goldens do), plus image, digest, driver, overlay and job provenance.Alternatives considered.
enable_autotune=Falsewould drop the worker-autotune and tuned-graph-capture path that this test was restored to cover.nvfp4_gemm_config) would be a public-API change and belongs in its own PR.Test Coverage
examples/visual_gen/test_visual_gen_ltx2.py::test_ltx2_cuda_graph_trtllm_backendruns only in B200 post-merge (DGX_B200-PyTorch-Post-Merge-1/2), so a plain/bot rundoes not select it. It was validated with the real pytest on B200 outside CI:nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc5.dev202611190007(digestsha256:a73ba957…, source 1f3b31b).tensorrt_llm/_torch/custom_ops/torch_custom_ops.pyfrom this PR's base 9d39212, which includes [https://nvbugs/7003734][fix] Preserve NVFP4 activation row and scale layouts #21220.tmp_path, autotuner cache, Inductor cache and Triton cache.00d73585…every time('cutlass',)Residual risks.
Links
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code