Repository navigation
[TRTLLM-17822][refactor] Move modeling_v2 out of _torch/_experimental - #22286
Conversation
|
/bot run --disable-fail-fast |
142e688 to
8a7eee6
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #84956 [ run ] triggered by Bot. Commit: |
|
/bot run --add-multi-gpu-test --extra-stage "GR100-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1" |
|
PR_Github #84960 [ run ] triggered by Bot. Commit: |
|
PR_Github #84956 [ run ] completed with state |
|
PR_Github #84960 [ run ] completed with state
|
8a7eee6 to
2b0e1e6
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #84986 [ run ] triggered by Bot. Commit: |
|
PR_Github #84986 [ run ] completed with state |
Follows the NVIDIA#22286 move of modeling_v2 out of _torch/_experimental: the four kernel changes carry over to the new paths unchanged, and the two source-scan tests drop the _experimental path element. No conflicts. Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; the five kernel files of this PR follow the move, no other change. Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
… in GLM-5.2 SM107 low-M split-K GEMMs _low_m_splitk.py released its accumulator TMEM through a parity-1 wait on the allocation barrier that the MMA warp reached without ever observing phase 0. If the MMA warp's second arrival lands before all epilogue warps have made their first arrival, phase 0 is still open, the parity-1 wait names the preceding phase and returns at once, and the tcgen05.dealloc races the epilogue warps' tcgen05.ld. The MMA warp now waits for phase 0 before its release arrival, the shape FlashInfer adopted in flashinfer-ai/flashinfer#7443 for the kernel this file is adapted from. The split-K mailbox waits in _low_m_splitk.py and _low_m_splitk_fused.py, and the amax pair exchange of the indexer q-tail epilogue, used the default CTA-scope acquire while the barriers are completed by peer CTAs' st.async complete-tx, a release at cluster scope. Under the PTX memory consistency model those pairs do not synchronize, so the reads after the wait were unordered against the peers' stores. A shared helper now polls the wait with mbarrier.try_wait.parity.relaxed.cluster and follows it with one fence.acquire.sync_restrict::shared::cluster.cluster, the PTX ISA's acquire pattern for shared memory. An acquire.cluster try-wait would make ptxas emit an L1 invalidate after every poll; the fence adds no instruction. Both forms are native in nvidia-cutlass-dsl 4.8.0 (cutlass.experimental.primitives), so the helper needs no inline PTX. A CPU-only source-level test pins both orderings, which an accuracy test cannot observe: the MMA warp's phase-0 wait between its two arrivals before the parity-1 wait and the TMEM dealloc, the epilogue's fenced accumulator load before its release arrival, and the four cross-CTA mailbox and amax waits going through the shared helper after the st.async scatter and before the payload reads. Its eight checks pass on this branch; on main seven fail. Rebased onto main after NVIDIA#22286 moved modeling_v2 to tensorrt_llm/_torch/modeling_v2. Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
…5.3-Flash tests modeling_v2 moved out of _torch/_experimental in NVIDIA#22286. Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
…/modeling_v2 modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location. Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
… static EPLB modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location. Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
… static EPLB modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location. Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
… online EPLB modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; update this PR's references to the new location and restore the ModelConfig import the merge dropped. Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
Move the added family sources out of _experimental and rewrite their imports, references and source-check paths after NVIDIA#22286. Keep the no-V1-import contract and adjust the relative-import probe for the shorter package path. Signed-off-by: Xin Guan <294044352+xguannv@users.noreply.github.com>
…er-trtllm-gen The update brings origin/main's Qwen3.5 GB300 round 2 (NVIDIA#21997), which builds new decode and context features on the FlashInfer trtllm-gen FMHA library this branch removes. They are carried over to the native trtllm-gen path: - Attention-DP decode with a minimum KV split per sequence: GqaDecodeFmha plans the prepared-query generation for splits x KV heads x batch SMs through the new FmhaParams.multi_processor_count, which the native generation uses for kernel selection and its multi-CTA scratch size. - context_fmha_head_dim_per_stage_kv: the native context passes the head dim per K/V stage to the trtllm-gen runner when that staged kernel is precompiled. - context_last_rows_only: in a context-only step with paged context FMHA, the native context attends from each request's last row only and zeros the other rows. - KV-replica: main's q all-gather in the RoPE kernel, gated E4M3 output and new exchange channels are kept on the native segmented / fused decode. FlashInfer-only pieces are dropped: the private decode-op wrapper and its signature probe, the segment softmax-stats workspace extension and module preloading, and their tests. The MiniMax-M3 drafter and dense decode keep this branch's native versions under the relocated modeling_v2 package (NVIDIA#22286); DFlash reads its page sizes from the native block decode. Signed-off-by: yihwang-nv <yihwang@nvidia.com>
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Correct the relative built-in fallback import for the new package depth. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Correct the relative built-in fallback import for the new package depth. Use upstream first-token completion handling and adapt the Beam handoff fixture; preserve reviewed cancellation and cache-identity behavior. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable. Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
…d review round - Move the target and its tests from _torch/_experimental/modeling_v2 to _torch/modeling_v2 after NVIDIA#22286. - ModelConfig: allow ssm_state_pool_geometry to be set on a frozen config (Runner.create_kv_cache_manager passes ModelConfig.from_pretrained); regression test added. - FMHA/attention tests: the last q-tile split depends on the GPU's SM count; check skipped leading rows only where the split runs. - Deployment guide: list bf16 as a selection condition; keeping mamba_ssm_cache_dtype=int8 without the target fails at startup. Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Important
Reviewers: only commit 2 (8 files) and the script that produces commit 1 need careful review. Commit 1 (about 2,000 files) is generated by that script and can be verified by rerunning it on the PR's base.
Authors of open modeling_v2 PRs: after this merges, rebase and run the migration commands.
Description
Moves
tensorrt_llm/_torch/_experimental/modeling_v2totensorrt_llm/_torch/modeling_v2and removes the_experimentalpackage, which held nothing else. Every reference to the old location is rewritten: imports, module-name strings, test paths, CBTS rules, agent-flow, CMake,setup.py,pyproject.toml, the SonarQube exclusion injenkins/TensorRT_LLM_PLC.groovyandCODEOWNERS.There is no behavior change. The
modeling_v2LLM argument, its default and itsprototypestatus are unchanged, and the pyrefly and SonarQube exclusions still cover the package at its new path. Test file paths do not change, so no test list or waive entry changes.The PR has two commits, best reviewed separately:
git mv, one rewrite of the old location, and the repository's pre-commit auto-fixes. The new module name sorts differently, so ruff re-sorts and re-wraps imports in about 350 files; an AST comparison shows those files differ from the un-fixed rewrite only in the order and wrapping of import statements. To check this commit, run the script on the base commit and compare._native.pylooked up the family libraries atPath(__file__).parents[3] / "libs", which after the move is one level abovetensorrt_llm. Nothing would have failed: every native family would look uninstalled,autowould fall back to the built-in model, and the library load test skips when it finds no library. It is nowparents[2], andtest_modeling_v2_native_loader.pyasserts that_LIBS_DIRistensorrt_llm/libs.cpp/and.github/lose one../._experimentalcomponent is dropped.modeling_v2package docstring carries what the removed_experimental/__init__.pysaid: nothing in the package is public API, andmodel_loader.pyandmodeling_auto.pyimport the package whatever the mode (only targets wait for routing).test_modeling_v2_package_location.pyin l0_cpu: the package istensorrt_llm/_torch/modeling_v2, no source file is left undertensorrt_llm/_torch/_experimental/modeling_v2, and no file outsidedocs/and3rdparty/names that location, as a path, a module name, a relative import or separate path components. Release notes indocs/may still name the old location.The script that produces commit 1 and how to verify it: see this comment.
Rebasing an open PR that touches modeling_v2
See this comment.
After pulling this change
--clean_wheel: an incrementalscripts/build_wheel.pydoes not remove deleted files from its staging tree, so it would package the old_torch/_experimental/modeling_v2next to the new one.trtllm_modeling_v2agent flow from a checkout on the same side of this change as the TensorRT-LLM checkout it drives; the flow names the package directory.setup.py's build-skew warning lists every familycsrcfile as changed when the precompiled artifacts predate this change; it is advisory.Test Coverage
Static checks on this branch: every absolute import of
tensorrt_llm._torch.modeling_v2(6,847) and every module-name string resolves to a file, no relative import leaves the package, every relative Markdown link in the package andcpp/tensorrt_llm/modeling_v2resolves, CMake still discovers the three families with native code (glm_moe_dsa, gpt_oss, kimi_k3_vl), and no tracked file names the old location.Before vs after the move on main bf069a7 (the move regenerated on that commit, and that commit's L0_PostMerge wheel):
-m cpu_only: 1,655 passed before and 1,658 after, the difference being the 3 new tests.Full unit run, before vs after on bf069a7: all 688 test files under
tests/unittest/_torch/modeling_v2plus the five other files that import or name the package. Each directory runs in its own pytest process with CI's flags (--timeout, waives, periodic JUnit). A first pass gives each process one GPU; files with a test skipped for want of GPUs then run again with every GPU of the node. Every test has the same outcome before and after; the only difference is the 3 new tests:The failures B200 and B300 show (identical before and after) are tests that need MNNVL, which DGX nodes lack and which pass on GB200 / GB300, and sm_107 tests that fail rather than skip on sm_100 / sm_103.
A wheel built from this branch (
scripts/build_wheel.py, CUDA archs 100 and 103), installed and tested on B200 and B300:_torch/modeling_v2that the main wheel ships under_torch/_experimental/modeling_v2, no path containing_experimental, and the three family libraries intensorrt_llm/libs, which carry the same targets as main's;test_every_installed_family_library_loadspasses (not skipped) and loads all three libraries; the routing tests and the tests that call the family libraries pass;modeling_v2="require"accuracy tests, one or more per target this hardware runs:r1_0528_nvfp4__sm_10x__defaulttest_gsm8k[dep4_moetp4]glm_5_3_flash_nvfp4__sm_100__defaulttest_gsm8k,test_gsm8k_mtp_vanillaglm_5_3_flash_nvfp4__sm_100__default_mtp_fusedtest_gsm8k_mtpglm_5_2_nvfp4__sm_103__tep4test_mtp3_gsm8k_and_acceptancegpt_oss_120b__sm_10x__defaulttest_gsm8k[tp1],test_gsm8k[tp4_ep4]minimax_m3_nvfp4__sm_103__tp4test_gsm8kminimax_m3_nvfp4__sm_103__tp8test_gsm8kqwen3_8_flash_next_nvfp4__sm_103__defaulttest_tp1_gsm8k,test_tp1_mtp1_gsm8k_mmmu_and_video,test_tp1_mtp3_gsm8k_and_long_context,test_mtp3_gsm8k_and_long_contextNegative controls: with the rewrite applied but the package left at the old location, 74 tests that pass after the move stop passing, so the tests that read the source do check the location.
test_modeling_v2_package_location.pyfails on the state of the generated commit alone (the two qwen4_exp paths split into components), on a source file under the old directory, and on each form of import of the old location; it passes on a checkout that keeps empty directories,__pycache__and build outputs there, and on a mention indocs/; it reads the files the same way in a source snapshot without.git.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.