Skip to content

[TRTLLM-17822][refactor] Move modeling_v2 out of _torch/_experimental - #22286

Merged
ruocheng-nv merged 2 commits into
NVIDIA:mainfrom
ruocheng-nv:refactor/mv2-drop-experimental-prefix
Nov 17, 2026
Merged

ruocheng-nv merged 2 commits into
NVIDIA:mainfrom
ruocheng-nv:refactor/mv2-drop-experimental-prefix

Conversation

@ruocheng-nv

@ruocheng-nv ruocheng-nv commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Important

Reviewers: only commit 2 (8 files) and the script that produces commit 1 need careful review. Commit 1 (about 2,000 files) is generated by that script and can be verified by rerunning it on the PR's base.

Authors of open modeling_v2 PRs: after this merges, rebase and run the migration commands.

Description

Moves tensorrt_llm/_torch/_experimental/modeling_v2 to tensorrt_llm/_torch/modeling_v2 and removes the _experimental package, which held nothing else. Every reference to the old location is rewritten: imports, module-name strings, test paths, CBTS rules, agent-flow, CMake, setup.py, pyproject.toml, the SonarQube exclusion in jenkins/TensorRT_LLM_PLC.groovy and CODEOWNERS.

There is no behavior change. The modeling_v2 LLM argument, its default and its prototype status are unchanged, and the pyrefly and SonarQube exclusions still cover the package at its new path. Test file paths do not change, so no test list or waive entry changes.

The PR has two commits, best reviewed separately:

  1. Move modeling_v2 to tensorrt_llm/_torch/modeling_v2 (generated) has no hand edits: git mv, one rewrite of the old location, and the repository's pre-commit auto-fixes. The new module name sorts differently, so ruff re-sorts and re-wraps imports in about 350 files; an AST comparison shows those files differ from the un-fixed rewrite only in the order and wrapping of import statements. To check this commit, run the script on the base commit and compare.
  2. Fix what the modeling_v2 move does not rewrite is hand-written (8 files):
    • _native.py looked up the family libraries at Path(__file__).parents[3] / "libs", which after the move is one level above tensorrt_llm. Nothing would have failed: every native family would look uninstalled, auto would fall back to the built-in model, and the library load test skips when it finds no library. It is now parents[2], and test_modeling_v2_native_loader.py asserts that _LIBS_DIR is tensorrt_llm/libs.
    • The package README's relative links to cpp/ and .github/ lose one ../.
    • Two qwen4_exp source checks build the kernel path from components; the _experimental component is dropped.
    • The modeling_v2 package docstring carries what the removed _experimental/__init__.py said: nothing in the package is public API, and model_loader.py and modeling_auto.py import the package whatever the mode (only targets wait for routing).
    • New test_modeling_v2_package_location.py in l0_cpu: the package is tensorrt_llm/_torch/modeling_v2, no source file is left under tensorrt_llm/_torch/_experimental/modeling_v2, and no file outside docs/ and 3rdparty/ names that location, as a path, a module name, a relative import or separate path components. Release notes in docs/ may still name the old location.

The script that produces commit 1 and how to verify it: see this comment.

Rebasing an open PR that touches modeling_v2

See this comment.

After pulling this change

  • Build the wheel once with --clean_wheel: an incremental scripts/build_wheel.py does not remove deleted files from its staging tree, so it would package the old _torch/_experimental/modeling_v2 next to the new one.
  • Run the trtllm_modeling_v2 agent flow from a checkout on the same side of this change as the TensorRT-LLM checkout it drives; the flow names the package directory.
  • setup.py's build-skew warning lists every family csrc file as changed when the precompiled artifacts predate this change; it is advisory.

Test Coverage

  • Static checks on this branch: every absolute import of tensorrt_llm._torch.modeling_v2 (6,847) and every module-name string resolves to a file, no relative import leaves the package, every relative Markdown link in the package and cpp/tensorrt_llm/modeling_v2 resolves, CMake still discovers the three families with native code (glm_moe_dsa, gpt_oss, kimi_k3_vl), and no tracked file names the old location.

  • Before vs after the move on main bf069a7 (the move regenerated on that commit, and that commit's L0_PostMerge wheel):

    • Collection on a B200: the modeling_v2 accuracy tests collect the same 85 IDs.
    • The l0_cpu modeling_v2 entries and the three other test files, -m cpu_only: 1,655 passed before and 1,658 after, the difference being the 3 new tests.
    • Importing every module of the package one at a time gives the same result before and after, on CPU and on a B200.
    • The agent-flow suite, run as CPU-AgentFlow-UnitTest runs it: 1,852 passed before and after.
    • The CBTS unit tests: 1,007 passed before and after.
  • Full unit run, before vs after on bf069a7: all 688 test files under tests/unittest/_torch/modeling_v2 plus the five other files that import or name the package. Each directory runs in its own pytest process with CI's flags (--timeout, waives, periodic JUnit). A first pass gives each process one GPU; files with a test skipped for want of GPUs then run again with every GPU of the node. Every test has the same outcome before and after; the only difference is the 3 new tests:

    Hardware Tests before / after Different outcomes
    8x B200 14,016 / 14,019 0
    8x B300 17,482 / 17,485 0
    GB200 NVL72, one tray: kimi_k3_vl, comm and MNNVL tests 2,925 / 2,925; all pass, 1 skipped (8 ranks on one 4-GPU tray) 0
    GB300 NVL72, one tray: the 44 modeling_v2 entries of l0_gb300 and l0_gb300_multi_gpus 2,322 / 2,322; all pass, 4 skipped (3 need 8 GPUs on one node) 0

    The failures B200 and B300 show (identical before and after) are tests that need MNNVL, which DGX nodes lack and which pass on GB200 / GB300, and sm_107 tests that fail rather than skip on sm_100 / sm_103.

  • A wheel built from this branch (scripts/build_wheel.py, CUDA archs 100 and 103), installed and tested on B200 and B300:

    • it ships the same 1,190 files under _torch/modeling_v2 that the main wheel ships under _torch/_experimental/modeling_v2, no path containing _experimental, and the three family libraries in tensorrt_llm/libs, which carry the same targets as main's;
    • test_every_installed_family_library_loads passes (not skipped) and loads all three libraries; the routing tests and the tests that call the family libraries pass;
    • modeling_v2="require" accuracy tests, one or more per target this hardware runs:
    Target Hardware Tests Result
    deepseek_v3 r1_0528_nvfp4__sm_10x__default B300 test_gsm8k[dep4_moetp4] 94.31 (ref 94.24)
    glm5_next glm_5_3_flash_nvfp4__sm_100__default B200 test_gsm8k, test_gsm8k_mtp_vanilla 90.56, 92.76 (ref 92.87)
    glm5_next glm_5_3_flash_nvfp4__sm_100__default_mtp_fused B200 test_gsm8k_mtp 92.80, acceptance length 3.39
    glm_moe_dsa glm_5_2_nvfp4__sm_103__tep4 B300 test_mtp3_gsm8k_and_acceptance passed
    gpt_oss gpt_oss_120b__sm_10x__default B200, B300 test_gsm8k[tp1], test_gsm8k[tp4_ep4] 91.13, 89.92, 89.92 (ref 90.30)
    minimax_m3 minimax_m3_nvfp4__sm_103__tp4 B300 test_gsm8k 91.24 (ref 88.70)
    minimax_m3 minimax_m3_nvfp4__sm_103__tp8 B300 test_gsm8k 89.42 (ref 88.70)
    qwen4_exp qwen3_8_flash_next_nvfp4__sm_103__default B300 test_tp1_gsm8k, test_tp1_mtp1_gsm8k_mmmu_and_video, test_tp1_mtp3_gsm8k_and_long_context, test_mtp3_gsm8k_and_long_context all passed
  • Negative controls: with the rewrite applied but the package left at the old location, 74 tests that pass after the move stop passing, so the tests that read the source do check the location. test_modeling_v2_package_location.py fails on the state of the generated commit alone (the two qwen4_exp paths split into components), on a source file under the old directory, and on each form of import of the old location; it passes on a checkout that keeps empty directories, __pycache__ and build outputs there, and on a mention in docs/; it reads the files the same way in a source snapshot without .git.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@ruocheng-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@ruocheng-nv
ruocheng-nv force-pushed the refactor/mv2-drop-experimental-prefix branch from 142e688 to 8a7eee6 Compare November 15, 2026 07:38
@ruocheng-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84956 [ run ] triggered by Bot. Commit: 8a7eee6 Link to invocation

@ruocheng-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --add-multi-gpu-test --extra-stage "GR100-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GB300-4_GPUs-PyTorch-Post-Merge-3,GB300-8_GPUs-2_Nodes-PyTorch-SingleNvlinkDomain-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84960 [ run ] triggered by Bot. Commit: 8a7eee6 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84956 [ run ] completed with state ABORTED. Commit: 8a7eee6

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84960 [ run ] completed with state SUCCESS. Commit: 8a7eee6
/LLM/main/L0_MergeRequest_PR pipeline #70316 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@ruocheng-nv
ruocheng-nv force-pushed the refactor/mv2-drop-experimental-prefix branch from 8a7eee6 to 2b0e1e6 Compare November 15, 2026 14:07
@ruocheng-nv

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84986 [ run ] triggered by Bot. Commit: 2b0e1e6 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #84986 [ run ] completed with state SUCCESS. Commit: 2b0e1e6
/LLM/main/L0_MergeRequest_PR pipeline #70338 completed with status: 'SUCCESS'

CI Report

Link to invocation

bastianhagedorn added a commit to bastianhagedorn/TensorRT-LLM that referenced this pull request Nov 17, 2026
Follows the NVIDIA#22286 move of modeling_v2 out of _torch/_experimental: the four
kernel changes carry over to the new paths unchanged, and the two source-scan
tests drop the _experimental path element. No conflicts.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
bastianhagedorn added a commit to bastianhagedorn/TensorRT-LLM that referenced this pull request Nov 17, 2026
modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; the five kernel
files of this PR follow the move, no other change.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
bastianhagedorn added a commit to bastianhagedorn/TensorRT-LLM that referenced this pull request Nov 17, 2026
… in GLM-5.2 SM107 low-M split-K GEMMs

_low_m_splitk.py released its accumulator TMEM through a parity-1 wait on
the allocation barrier that the MMA warp reached without ever observing
phase 0. If the MMA warp's second arrival lands before all epilogue warps
have made their first arrival, phase 0 is still open, the parity-1 wait
names the preceding phase and returns at once, and the tcgen05.dealloc
races the epilogue warps' tcgen05.ld. The MMA warp now waits for phase 0
before its release arrival, the shape FlashInfer adopted in
flashinfer-ai/flashinfer#7443 for the kernel this file is adapted from.

The split-K mailbox waits in _low_m_splitk.py and _low_m_splitk_fused.py,
and the amax pair exchange of the indexer q-tail epilogue, used the default
CTA-scope acquire while the barriers are completed by peer CTAs' st.async
complete-tx, a release at cluster scope. Under the PTX memory consistency
model those pairs do not synchronize, so the reads after the wait were
unordered against the peers' stores. A shared helper now polls the wait
with mbarrier.try_wait.parity.relaxed.cluster and follows it with one
fence.acquire.sync_restrict::shared::cluster.cluster, the PTX ISA's acquire
pattern for shared memory. An acquire.cluster try-wait would make ptxas
emit an L1 invalidate after every poll; the fence adds no instruction. Both
forms are native in nvidia-cutlass-dsl 4.8.0 (cutlass.experimental.primitives),
so the helper needs no inline PTX.

A CPU-only source-level test pins both orderings, which an accuracy test
cannot observe: the MMA warp's phase-0 wait between its two arrivals before
the parity-1 wait and the TMEM dealloc, the epilogue's fenced accumulator
load before its release arrival, and the four cross-CTA mailbox and amax
waits going through the shared helper after the st.async scatter and before
the payload reads. Its eight checks pass on this branch; on main seven fail.

Rebased onto main after NVIDIA#22286 moved modeling_v2 to tensorrt_llm/_torch/modeling_v2.

Signed-off-by: Bastian Hagedorn <bhagedorn@nvidia.com>
ruocheng-nv added a commit to ruocheng-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
…5.3-Flash tests

modeling_v2 moved out of _torch/_experimental in NVIDIA#22286.

Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
ruocheng-nv added a commit to ruocheng-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
…/modeling_v2

modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location.

Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
ruocheng-nv added a commit to ruocheng-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
… static EPLB

modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location.

Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
ruocheng-nv added a commit to ruocheng-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
… static EPLB

modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; move this PR's new files and references to the new location.

Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
ruocheng-nv added a commit to ruocheng-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
… online EPLB

modeling_v2 moved out of _torch/_experimental in NVIDIA#22286; update this PR's references to the new location and restore the ModelConfig import the merge dropped.

Signed-off-by: Ruocheng Jia <ruochengj@nvidia.com>
xguannv added a commit to xguannv/TensorRT-LLM that referenced this pull request Nov 17, 2026
Move the added family sources out of _experimental and rewrite their imports, references and source-check paths after NVIDIA#22286. Keep the no-V1-import contract and adjust the relative-import probe for the shorter package path.

Signed-off-by: Xin Guan <294044352+xguannv@users.noreply.github.com>
yihwang-nv added a commit to yihwang-nv/TensorRT-LLM that referenced this pull request Nov 17, 2026
…er-trtllm-gen

The update brings origin/main's Qwen3.5 GB300 round 2 (NVIDIA#21997), which builds new
decode and context features on the FlashInfer trtllm-gen FMHA library this branch
removes. They are carried over to the native trtllm-gen path:

- Attention-DP decode with a minimum KV split per sequence: GqaDecodeFmha plans the
  prepared-query generation for splits x KV heads x batch SMs through the new
  FmhaParams.multi_processor_count, which the native generation uses for kernel
  selection and its multi-CTA scratch size.
- context_fmha_head_dim_per_stage_kv: the native context passes the head dim per
  K/V stage to the trtllm-gen runner when that staged kernel is precompiled.
- context_last_rows_only: in a context-only step with paged context FMHA, the native
  context attends from each request's last row only and zeros the other rows.
- KV-replica: main's q all-gather in the RoPE kernel, gated E4M3 output and new
  exchange channels are kept on the native segmented / fused decode.

FlashInfer-only pieces are dropped: the private decode-op wrapper and its signature
probe, the segment softmax-stats workspace extension and module preloading, and
their tests. The MiniMax-M3 drafter and dense decode keep this branch's native
versions under the relocated modeling_v2 package (NVIDIA#22286); DFlash reads its page
sizes from the native block decode.

Signed-off-by: yihwang-nv <yihwang@nvidia.com>
Mgluhovskoi added a commit to Mgluhovskoi/TensorRT-LLM that referenced this pull request Nov 17, 2026
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Correct the relative built-in fallback import for the new package depth. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Mgluhovskoi added a commit to Mgluhovskoi/TensorRT-LLM that referenced this pull request Nov 17, 2026
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Mgluhovskoi added a commit to Mgluhovskoi/TensorRT-LLM that referenced this pull request Nov 17, 2026
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Correct the relative built-in fallback import for the new package depth. Use upstream first-token completion handling and adapt the Beam handoff fixture; preserve reviewed cancellation and cache-identity behavior. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Mgluhovskoi added a commit to Mgluhovskoi/TensorRT-LLM that referenced this pull request Nov 19, 2026
Follow upstream NVIDIA#22286: migrate added Beam modules, imports, documentation and test references to tensorrt_llm._torch.modeling_v2. Preserve the package-location guard and current upstream test selection. Validation: 83 tracked package/CI-selection checks and 90 host-adapted component checks per branch; pre-commit passed. Native GPU/distributed qualification remains separate. Advisory Pyrefly unavailable.

Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
nvchenghaoz added a commit to nvchenghaoz/TensorRT-LLM that referenced this pull request Nov 20, 2026
…d review round

- Move the target and its tests from _torch/_experimental/modeling_v2 to
  _torch/modeling_v2 after NVIDIA#22286.
- ModelConfig: allow ssm_state_pool_geometry to be set on a frozen config
  (Runner.create_kv_cache_manager passes ModelConfig.from_pretrained);
  regression test added.
- FMHA/attention tests: the last q-tile split depends on the GPU's SM count;
  check skipped leading rows only where the split runs.
- Deployment guide: list bf16 as a selection condition; keeping
  mamba_ssm_cache_dtype=int8 without the target fails at startup.

Signed-off-by: Chenghao Zhang <211069071+nvchenghaoz@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.