Repository navigation
[None][feat] Nemotron-3 Ultra: modeling_v2 targets (NVFP4 sm_107, BF16 sm_100) and disagg serving fixes - #20809
Draft
nv-guomingz wants to merge 41 commits into
Draft
nv-guomingz wants to merge 41 commits into
nv-guomingz wants to merge 41 commits into
Conversation
nv-guomingz
force-pushed
the
user/guomingz/mv2-nemotron-ultra-sm107
branch
3 times, most recently
from
November 16, 2026 01:34
99fc3e1 to
dd4ce8f
Compare
Add the nemotron_h family and its first target, the Nemotron-3 Ultra 550B-A55B BF16 release on sm_100 (GB200): 8 ranks, tensor-parallel attention, mamba and shared expert, expert-parallel routed experts. Every GEMM is cublas_mm over bf16 weights, the routed experts run through a CUTLASS fused_moe with Relu2 (non-gated FC1), the KV cache is bf16 and the SSM state fp32. Ported from user/guomingz/mv2-nemotron-ultra-sm107 onto the current modeling_v2 conventions: - Routing: models/nemotron_h/routing.py registers its route with @routes("NemotronHForCausalLM") and is imported from models/__init__.py. The route checks every requirement the target has, in order: the checkpoint shape and quantization, sm_100, the parallel layout (tp8 with moe_ep 8, no pp/cp/moe cluster, two trays of four GPUs), then the engine choices (no speculative decoding, bf16, beam width 1, KV cache manager V2, block reuse off, no EPLB, the CUTLASS MoE backend, no piecewise prefill graphs, a bf16 KV cache and an fp32 SSM state). - Family ops under models/nemotron_h/ops/ instead of new catalog entries, since only this family calls them: ssm/causal_conv1d_fwd, ssm/causal_conv1d_update, ssm/mamba_chunk_scan_combined, ssm/selective_state_update, norm/rms_norm_gated_grouped and moe/fused_moe (DeepSeek's op plus the Relu2 activation and the Ultra cells). ARCHS is {SM_100}; the qwen4_exp cells of the conv entries are not carried, and the conv-state slot offsets are 64-bit since NVIDIA#21684. - The plain sum over the group calls the catalog comm/allreduce entry with op NONE through allreduce_group_args (AUTO in one node, NCCL across nodes), in place of the branch's own allreduce entry. - thop_attention call sites use the renamed workspace and spec_decoding_position_offsets keywords (NVIDIA#17558). - Tests: the family routing test, the family op tests under unittest/.../models/nemotron_h/ops/, the tep8 weight-table test (cpu_only) and a per-target GSM8K gate whose engine runs inside scoped_env_overrides. The BF16 GSM8K reference is the built-in implementation's 94.769. - Lists: l0_cpu (routing, tep8 directory), l0_b200 (family ops) and qa/llm_function_multinode (tep8 GSM8K). Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ltra BF16 tep8 target Across nodes on an NVL72-class system (aarch64, fabric handles) the tep8 target runs the sum over the group, the residual add and the next block's RMSNorm as one MNNVL op, instead of NCCL followed by the fused add + RMSNorm. Before the rebase, on two GB200 nodes (8 ranks, trtllm-bench c=32, 1k/1k), that was 2473 tok/s against 2087 with NCCL; GSM8K 94.807 (reference 94.769). Ported onto the current catalog: - The fused path calls the catalog comm/mnnvl_allreduce entry (the same torch.ops.trtllm.mnnvl_fusion_allreduce call with RESIDUAL_RMS_NORM; the residual now comes before the norm weight) instead of the branch's own mnnvl_fusion_allreduce entry, which is not carried, nor is its op matrix. - The workspace is built once at load by mnnvl_workspace_args for max_num_tokens rows, instead of growing per forward. The uint32 limit the kernels record buffer sizes in is kept: a budget past it raises, and every rank falls back to NCCL, as a failed workspace setup does. - The is_mnnvl gate, the allreduce_strategy opt-out and the log line are unchanged. - _bind_cache_manager asserts a bf16 KV pool, a bf16 conv state and an fp32 SSM state; a cpu_only test covers it. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…et at max_batch_size 256 The tep8 target was gated at max_batch_size 32 and its ops certified up to 32-row decode batches. At max_batch_size 256 the engine's recurrent-state pool holds 271 slots (35 at 32), and a layer's fp32 SSM view has slot stride 48 * 32 * 64 * 128 = 12.6M elements, so the slots from 171 on lie past 2^31 elements of the base. - Family ops: 256-row decode cells and the 271-slot pool for selective_state_update and causal_conv1d_update (into the last slot, past 2^31 elements), causal_conv1d_fwd and mamba_chunk_scan_combined over 256 sequences of one 8192-token step, the gated grouped norm, and fused_moe at 256 rows (ranks 0 and 7). - The target logs the depth of the pools it binds. - The tep8 GSM8K gate runs at max_batch_size 256 (94.617 before the rebase, reference 94.769, threshold 91.566). Not carried here: this commit's cells on the shared catalog entries (thop_attention, cublas_mm, noaux_tc_op, the flashinfer norms) move to the entry tests' TARGET_CELLS in a later commit, and its allreduce/MNNVL cells are left for a separate shared-catalog change. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
… 2 x moe_tp 4) The tep8 target splits the 512 routed experts into eight 64-expert windows, and the all-reduce after every MoE block waits for the rank that drew the most tokens. Two windows of four tensor-parallel ranks keep the balance of the all-tensor-parallel split and the prefill efficiency of the expert-parallel one. Before the rebase, on two GB200 nodes at max_batch_size 256, the built-in implementation's ep 2 x tp 4 layout gave 145.5 / 2515.1 / 5732.1 output tok/s at c=1 / 32 / 128 against 140.5 / 2366.6 / 5441.8 for ep 8. - models/nemotron_h/nemotron3_ultra_bf16__sm_100__tp8ep2: tep8's forward with the routed experts read as the engine's mapping lays them out for moe_ep 2 x moe_tp 4 (rank r runs window r % 2, each expert on its quarter r // 2 of the 5120-wide intermediate), fused_moe handed its tp coordinates, and the same catalog allreduce / mnnvl_allreduce calls as tep8. - Routing admits (world 8, no attention DP, moe_tp 4, moe_ep 2) with a bf16 KV cache; the family routing test covers the match and its near misses. - fused_moe (family op): six tp8ep2 cells, 1 to 8192 rows, across both windows and all four slices. - cpu_only weight-table and cache-bind tests in the target's unit directory, and the GSM8K gate TestModelingV2Nemotron3UltraBf16Sm100Tp8ep2 (94.579 before the rebase, reference 94.769, threshold 91.566). Not carried: the promotion of mnnvl_fusion_allreduce to the catalog (the targets call the catalog's mnnvl_allreduce) and its op matrix. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…otron-3 Ultra BF16 targets At high concurrency the Mamba2 decode step is a visible share of a decode step: before the rebase, at this rank's shapes (32 heads of 64, one group of 128 states, an fp32 state) FlashInfer's selective_state_update replayed 1.17x as fast as the Triton kernel at 32 rows, 1.31x at 128 and 1.36x at 256, and was the slower one below 32 rows. - New family op models/nemotron_h/ops/ssm/flashinfer_selective_state_update (ARCHS sm_100) over flashinfer.mamba.selective_state_update: the contract and reference of selective_state_update, cells at 32 to 256 rows on the engine's 35- and 271-slot fp32 pools, and the slot hazards is_valid refuses. Its test moves to unittest/.../models/nemotron_h/ops/ssm/. - Both BF16 targets run it in their decode step from 32 rows on, with bf16 copies of D and dt_bias. A mixed step's generation rows stay on the Triton kernel: FlashInfer's reads x, B and C through TMA and needs them 128-byte aligned, which rows past an odd number of context rows are not. Before the rebase, GSM8K at max_batch_size 256: tep8 94.617, tp8ep2 94.920 (reference 94.769). The op and its tuning ran under FlashInfer 0.6.18; main pins 0.7.2, so the op test and the decode throughput are to be re-run. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…tron-3 Ultra decode projections At one or two decode rows every dense projection reads its whole weight to produce a row or two. Before the rebase, on the rank's K = 8192 shapes FlashInfer's register-prefetch kernel (dense_bf16_gemm_direct) replayed 1.3-2.3x as fast as cublas_mm at one row, was still ahead at two rows and not from four on. - New family op models/nemotron_h/ops/gemm/flashinfer_direct_bf16_gemm (ARCHS sm_100): bf16 `mat_a @ mat_b` with mat_b column-major as cublas_mm reads it, one fixed tactic per certified shape (so ranks running a replicated projection get the same bits), the caller choosing PDL. Cells for the four K = 8192 projections at one and two rows; its test moves to unittest/.../models/nemotron_h/ops/gemm/. - Both BF16 targets run the mamba in_proj, qkv, the shared expert's up projection and fc1_latent through it in decode steps of up to two rows; the router stays on cublas_mm (fp32 logits). Before the rebase, tp8ep2 on two GB200 nodes: one request 154.1 against 146.0 output tok/s (time per output token 6.78 -> 6.39 ms), eight requests unchanged within noise; GSM8K tp8ep2 94.845, tep8 94.617 (reference 94.769). The tactic table was tuned under FlashInfer 0.6.18; main pins 0.7.2. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…p8ep2 target Agentic sessions carry tens of thousands of tokens of context, and at TP8 each rank stores a whole kv head per token: 6 KB per token per rank in bf16. Before the rebase, gen-only on two GB200 nodes at a 42k-token context, an fp8 pool gave +19% / +31% output tok/s at 64 / 128 requests over the bf16 pool. - tp8ep2 binds an fp8-e4m3 KV pool too (`kv_cache_config.dtype: fp8`) at unit kv scales and passes QM_FP8_KV_CACHE to both attention call sites from the bound pool. Over an fp8 pool both kv scales are one fp32 [1] tensor of ones allocated at load, the form main's thop_attention certifies (the branch passed None, which the native op drops); over a bf16 pool quant_mode stays 0 with no scales. Any other pool dtype is still refused when the pools are taken. - Routing admits an FP8 engine KV cache at tp8ep2; tep8 stays bf16-only. - The bind test certifies bf16 and fp8 (with the unit scale tensor) and still refuses NVFP4 / fp16 pools. - GSM8K gate test_gsm8k_fp8_kv, with a gsm8k.yaml entry for the BF16 release over an FP8 KV cache: the built-in implementation's 94.655 (the target measured 94.731 before the rebase). The thop_attention cells for this geometry over an e4m3 pool move to the entry test's TARGET_CELLS in the next commit. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…catalog entry tests The BF16 targets call five shared catalog entries with shapes no other target uses. On the source branch those shapes were cells inside the catalog sources (3c16610, e64f5a0, a023637, a3ad2b8, 037bd53); here they are TARGET_CELLS in the entries' tests, keyed by the target that calls them, and no catalog source changes. - attention/thop_attention: 8 q heads over one kv head at d128, no rope, over a bf16 pool (tep8: packed prefill, decode at 1 / 32 / 256 rows, a packed mixed step, paged-context prefill) and over an fp8-e4m3 pool at unit kv scales (tp8ep2, the same calls bar the packed mixed step, which reads context rows in bf16 and generation rows in e4m3). The paged harness takes an optional max_blocks_per_seq (default 16, so every existing cell is unchanged) for the 1100- and 1500-token histories, and the fp8 8:1 d128 cells are gated in an e4m3 split-softmax band derived in _compare_split_softmax; every other cell keeps _compare. - gemm/cublas_mm: every dense projection of a tp8 rank, and the fp32-out router, at 1, 32, 256 and 8192 rows. - moe/noaux_tc_op: the router at top-22 of 512, scale 5.0, fp32 logits, with synthetic biases in the BF16 release's ranges instead of checkpoint reads. _build takes optional logit_std and bias keys whose defaults reproduce the existing draws; the Ultra cells name a derived weights band (fp32 sums on both sides), every other cell keeps assert_close. - norm/flashinfer_rmsnorm and norm/flashinfer_fused_add_rmsnorm: (T, 8192) bf16, eps 1e-5, at 1, 32, 256 and 8192 rows. Not carried: the branch's Ultra hybrid-pool cells (attention rows on pool 1 of a real MambaHybridCacheManagerV2), which need its _UltraHybridEnv and a generalized check_caches in the shared paged harness. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
List the three nemotron_h paths -- the package, its unit tests and its accuracy tests -- as the Modeling V2 guide asks for a family with its own owners. A commit of its own: .github/CODEOWNERS is owned by infra-devs and oss-compliance and turns on the OSS-Compliance-Check stage, so it can move to a separate change if it blocks. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ep4, dep4) Add two modeling_v2 targets for NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 (NemotronHForCausalLM: Mamba2 + no-RoPE GQA + LatentMoE) on sm_107: tep4 (TP4 attention/mamba, EP4 MoE) and dep4 (attention DP, EP4 MoE). Ported from 3c16610 (with f62e195's switch of tep4 to a shared all-reduce) onto the current modeling_v2 conventions: - Routing admits the NVFP4 release (MIXED_PRECISION, FP8 KV) on sm_107 at (4 ranks, moe_ep 4) with attention DP off (tep4) or on (dep4), and requires an FP8 engine KV cache, an fp16 SSM state without stochastic rounding and the TRTLLM MoE backend, besides the checks the BF16 targets share. The family routing test covers both matches and the near misses. - Family ops under models/nemotron_h/ops/: gemm/cublas_scaled_mm, quantization/static_quantize_e4m3_per_tensor, and moe/fp4_block_scale_moe_relu2, which forwards every argument to the catalog fp4_block_scale_moe_runner and certifies Relu2 on top: its own fp32 reference (non-gated FC1, per-expert weight_scale_2, NVFP4 requantization of the relu2 output), the FC1 row-count guard and the Ultra cells. The catalog runner is unchanged. - tep4's plain sum calls the catalog comm/allreduce entry with op NONE through allreduce_group_args (in-tray IPC, AUTO); dep4 keeps the catalog allgather/reducescatter. Both pass the fp8 pool's kv scales as one fp32 [1] tensor of ones; the checkpoint's scales are 1.0, asserted at load. - TARGET_CELLS for both targets in the shared entry tests: thop_attention (16:1 and 64:2 d128 over an fp8 pool), cublas_mm (their bf16 projections) and fp4_quantize (the 2048-wide latent, and a static scale the batch outruns, with amax_headroom support in _build). - GSM8K gates in qa/llm_function_core.txt (4 Rubin GPUs; CI has no 4-GPU sm_107 stage). The family ops' ARCHS is {SM_100}, where their tests run in CI; SM_107 is to be added after a Rubin run. Before this rebase the targets measured GSM8K tep4 94.731 and dep4 94.769 on sm_107 (reference 94.655); they have not been re-run on the new base. Not carried: the promotions of noaux_tc_op, fused_moe and causal_conv1d to the catalog, the edits to deepseek_v3 and qwen4_exp, the _op.py and _router_index.py hunks, ARCHS and cells added to catalog sources, the README section, the claims-suite hunk, tep4's target-local allreduce and its op matrix, and the Ultra hybrid-pool thop_attention cells. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ed-comm launch Each MegaMoE launch returns a view of a combine buffer that the next launch overwrites, so multi-chunk forwards returned wrong rows. Clone every non-final chunk output; single-chunk forwards are unchanged. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…p to 32 Add the BF16 MegaMoE leaf (trtllm.cutedsl.mega_moe.none) with a non-gated ReLU2 epilogue and top_k up to 32, for Nemotron-3 Ultra BF16. NVFP4 is unchanged. Tested on SM100. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…r the Nemotron-3 Ultra BF16 tep8 target (part 1) moe_config.backend MEGAMOE_CUTEDSL or TRTLLM, asked for by name, runs the tep8 target's routed experts in the engine's MoE layer (create_moe) on the BF16 Relu2 leaf: trtllm.cutedsl.mega_moe.none or flashinfer.trtllm_gen.fused_moe.none. AUTO and CUTLASS keep the target's own fused_moe. Backend degradation is off and the leaf and expert window are checked at construction, so a declined request fails loudly. tp8ep2 keeps refusing both (MegaMoE is EP-only; moe_tp is 4 there). The weight table declares no routed experts on that path and feeds each engine layer its window. Stock calls are declared in UNCERTIFIED_GENERIC_CALLS, checked by a family test. CPU tests for routing, create_moe arguments and the weight feed; GSM8K variants for both backends in the QA multinode list. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…on DP, moe_ep 16) A modeling_v2 target for the Nemotron-3 Ultra BF16 release on 16 GB200 GPUs (four trays): attention data parallel, the routed experts expert-parallel in sixteen 32-expert windows, tp 16 / ep 16. This is the layout of the disaggregated serving tiers (ctx DEP16 + gen DEP16). - Structure from the NVFP4 dep4 target: every rank holds whole copies of the mamba, attention and shared-expert blocks, the router and both latent projections (about 63 GiB), runs its own requests, and adds each block's output straight to the residual. No all-reduce. - Compute from the BF16 tep8 target: bf16 GEMMs, FlashInfer's low-M kernel for the K = 8192 decode projections at one or two rows, FlashInfer's Mamba2 decode step from 32 rows, the Triton step below and on mixed steps. - Routed experts: always the engine's MoE layer (create_moe, Relu2 leaf), because its exchange between ranks is what this layout needs. CUTLASS and TRTLLM dispatch and combine the latent with the engine's all-to-all (NVLinkOneSided where available, as Modeling V1 runs this layout); MEGAMOE_CUTEDSL fuses dispatch, experts and combine. AUTO resolves to CUTLASS. Construction pins the leaf, the 32-expert window, attention DP and the presence of an exchange, and fails loudly otherwise. - Weights: everything but the experts whole; each engine MoE layer is fed its window as w1 = up_proj, empty w3, w2 = down_proj; lm_head whole. - Knobs read at load: TRTLLM_NEMOTRON_ULTRA_DEP16_LOWM_MAX_ROWS (0-2) and TRTLLM_NEMOTRON_ULTRA_DEP16_FI_SSU_MIN_ROWS. - Routing: BF16 on sm_100 at (world 16, attention DP, moe_tp 1, moe_ep 16) selects dep16 with the AUTO / CUTLASS / MEGAMOE_CUTEDSL / TRTLLM backends, a bf16 KV cache and an fp32 SSM state. Every other layout routes as before. Not yet run on GPUs. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…16 target - Routing: BF16 on sm_100 at 16 ranks with attention DP and moe_ep 16 selects dep16 under auto and require, with each of AUTO / CUTLASS / MEGAMOE_CUTEDSL / TRTLLM; near misses (tp16 without attention DP, moe_ep 8 x moe_tp 2, dep8, one 16-GPU node, vocab-parallel lm_head, fp8 KV, fp16 state, block reuse, EPLB, piecewise graphs, another MoE backend, speculative decoding, the NVFP4 release) fall back. The existing tep8 / tp8ep2 / NVFP4 cases are unchanged. - Weight table: every rank fills every declared weight once, reads every tensor whole, declares no routed expert; the engine feed is the rank's 32-expert window and tiles the 512 experts across the 16 ranks; qkv layout; leftover keys are exactly MTP, off-window experts and layers past the stack. - Engine MoE (create_moe stand-in): the create_moe arguments, one layer per MoE block for each backend, refusal of the wrong leaf / window / a layer without attention DP / an external-comm layer without an exchange / a resolved backend other than the request's; the load-time knobs' bounds; the bf16 KV / fp32 state pool checks. - The uncertified-generic-calls test now requires dep16 to declare its create_moe use, and l0_cpu runs the new directory. Run on a login node (no tensorrt_llm build there): the weight-table and uncertified-calls tests pass. Routing and engine-MoE tests need the wheel. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ep16 target The BF16 accuracy harness takes the layout from the subclass (ranks, MoE EP size, attention DP, the engine's max_num_tokens) and an optional MoE launch cap; tep8 and tp8ep2 run as before. dep16 runs on 16 GB200 GPUs with attention DP, 32 requests and 2048 tokens a rank: - test_gsm8k_megamoe_cutedsl: MegaMoE with moe_config.max_num_tokens 1024 (the disaggregated tiers' cap), so a full prefill step takes two launches; - test_gsm8k: AUTO, the CUTLASS leaf over the engine's all-to-all; - test_gsm8k_trtllm_moe: TRTLLM-Gen BF16 MoE over the all-to-all. All three are in qa/llm_function_multinode.yml (CI has no 16-GPU GB200 stage). test_modeling_v2_ci_selection passes on a login node. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…6 shapes dep16 runs every dense op unsharded; where that is a shape, layout or geometry no cell covered, add the cell (GPU tests): - cublas_mm: the nine dep16 projections (in_proj 8192 -> 35072, out_proj 16384 -> 8192, qkv 8192 -> 8704, o_proj, shared up / down 8192 <-> 10240, the latent pair, the fp32-out router) at 1, 32 and 4096 rows. - thop_attention: 64 q over 2 kv heads, d128, over the bf16 pool (packed and paged prefill, decode at 1 and 32 rows, a packed mixed step). - flashinfer_direct_bf16_gemm: the dep16 low-M shapes at one and two rows (N 35072, 8704, 10240); they run the kernel's default tactic, untimed. - flashinfer_selective_state_update: the 256-head / 8-group geometry at 1, 8 and 32 rows and with pad rows, over a 35-slot x 48-layer fp32 pool whose slots from 22 on lie past 2^31 elements. The test takes the geometry from the cell (nheads / ngroups) and requires those cells to touch such slots. - selective_state_update: the dep16 operands together (256 heads, fp32 pool, fp32 D and dt_bias) at 31 rows; the geometry was already certified by dep4. The SSM conv, scan and gated-norm ops were already certified at this geometry by the dep4 cells. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…16 target runs with The rank-0 load line now names, for MEGAMOE_CUTEDSL, the combine wire format (MEGAMOE_COMBINE_FORMAT), whether the tactic sweep is on (MEGAMOE_TACTIC_AUTOTUNE) and the per-rank token capacity of its largest launch bucket (min of moe_config.max_num_tokens and max_num_tokens), so a sweep's log records the point it measured. External-comm backends keep naming their communication strategy. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…kens a rank The MegaMoE gate now runs the disaggregated context tier's step budget, 4096 tokens a rank, with its 1024-row launch cap, so a full prefill step takes the four-launch multi-chunk path. The CUTLASS and TRTLLM gates stay at 2048, whose default MoE launch cap (32768) is the setting Modeling V1 ran this layout with. The harness takes a per-test max_num_tokens. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…easured The routing comment for the dep16 layout now says it was measured with the built-in implementation (the disaggregated tiers); the target's own GSM8K gate is in its accuracy directory. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
… BF16 dep16 target The disaggregated context tier reuses each conversation's prefix (enable_block_reuse with prompt-end Mamba snapshots), which every nemotron_h target refuses. With TRTLLM_NEMOTRON_ULTRA_DEP16_BLOCK_REUSE=1 routing admits it for dep16 only: the cache manager takes and restores the snapshots in the state pools, and the target's prefill already seeds the scan and the conv window from the slot wherever has_initial_states says so, as Modeling V1's Mamba2 mixer does (which has no snapshot code of its own). Not gated yet, hence opt-in; without the variable the refusal names it. Other targets keep refusing block reuse. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…MMs for the Ultra BF16 dep16 decode Decode on the dep16 generation tier is bound by weight bandwidth. Two knobs for it, off by default and read once at load: - TRTLLM_NEMOTRON_ULTRA_DEP16_MOE_FRONT_FIRST=1 issues the latency-bound MoE front (the router GEMM, fc1_latent) before the shared expert's GEMMs take the SMs: the same kernels on the same inputs (Modeling V1's TRTLLM_NEMOTRON_H_MOE_FRONT_FIRST). - TRTLLM_NEMOTRON_ULTRA_DEP16_SPLITK_MAX_ROWS=n (0 is off, at most 32) runs the dense projections named in TRTLLM_NEMOTRON_ULTRA_DEP16_SPLITK_PROJECTIONS (default: mamba in/out_proj, shared up, fc1/fc2_latent) through a new target op, ops/gemm/flashinfer_splitk_bf16_gemm.py (FlashInfer's split-K BF16 GEMM with fixed tactics), at up to n rows, where cuBLAS leaves SMs idle. Measured on 32 GB200 (disaggregated context dep16 with MegaMoE, generation dep16 with the TRTLLM MoE, SWE-like closed loop): front-first +0.6% / +0.9% tok/s/GPU at 256 / 512 agents (decode speed +3.4%), split-K on top +4.0% / +3.2% (decode speed +3.9%). GSM8K with both on and the TRTLLM MoE: 94.84 (94.62 without). The op's unit test holds it to cublas_mm's 4-ulp band at every dep16 cell, with bitwise repeats and CUDA graph replay; the target's CPU tests cover the knobs' parsing. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…on DP, moe_ep 32) A disaggregated deployment can run its context tier at DEP32 beside generation tiers at dep16 (1P2D). Routing picks a target by the engine's own parallel layout, so a 32-rank engine needs its own target: nemotron3_ultra_bf16__sm_100__dep32 is the dep16 target with the routed experts in thirty-two 16-expert windows. The expert window is the only difference (a weight layout); every non-expert shape a rank runs is dep16's. Routing admits (32 ranks, attention DP, moe_tp 1, moe_ep 32) on GB200 trays with dep16's KV cache algorithms and MoE backends (AUTO/CUTLASS, MEGAMOE_CUTEDSL, TRTLLM), and block reuse behind its own opt-in, TRTLLM_NEMOTRON_ULTRA_DEP32_BLOCK_REUSE, so a context tier at DEP32 and generation tiers at dep16 decide reuse independently. Tests: the target's CPU tests (engine MoE wiring, weight table), dep32 routing cases, its cells in the cublas_mm and thop attention entries (dep16's specs: the same shapes), and GSM8K gates (MegaMoE at 4096 tokens a rank with 1024-row launches, CUTLASS, TRTLLM MoE). The gates stay out of the test lists (_ACCURACY_NOT_LISTED_YET) until they are listed beside dep16's. Measured on 32 GB200 (eight trays, one NVLink domain): GSM8K MegaMoE 94.85 and CUTLASS 94.50. The MegaMoE number needs the router fence main's NVIDIA#21977 added in token_comm.py; without it the router's size broadcast drops routes at EP32. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ron-3 Ultra BF16 dep16/dep32 targets With prefill_cuda_graph_backend: breakable the engine captures the prefill body once per token bucket and replays it; anything not behind a graph break is frozen at capture time. The nemotron_h targets had no breaks, so a captured prefill replayed capture-time attention and Mamba metadata and produced NaN logits (the sampler's device assert on a context tier). dep16 and dep32 now run the two step-dependent parts of their prefill eagerly inside the graph through catalog/graph/eager_on_graph: the Mamba core (varlen causal conv1d and chunked scan for context rows, the state update for decode rows) and attention (thop). Each break reads the step's metadata from get_model_extra_attrs()["attention_metadata"] at replay, works on the live rows only and zeroes the padded tail of its fixed output; the dense GEMMs and the MoE layer between the breaks stay captured. tep8 and tp8ep2 have no breaks and now refuse a breakable capture with an error instead of computing NaN. eager_on_graph is admitted on SM100 as well as SM103. Gates (MegaMoE, buckets 1..4096, test_gsm8k_megamoe_cutedsl_bcg): dep16 94.92, dep32 94.69, no NaN; the eager_on_graph op test passes on SM100. On the disaggregated SWE-like workload the context tier gained nothing from it (tok/s/GPU -0.7% / -2.3% at 256 / 512 agents, higher TTFT), so it stays an engine option, off by default. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…and the dep16/dep32 BCG gates dep32's GSM8K gates have passed on 32 GB200 GPUs (eight trays in one NVLink domain): MegaMoE 94.85, MegaMoE with breakable prefill CUDA graphs 94.69, CUTLASS 94.50, TRTLLM MoE 94.92 (reference 94.77). List them in qa/llm_function_multinode.yml beside dep16's, together with dep16's breakable-graph gate (94.92), and drop the _ACCURACY_NOT_LISTED_YET exemption that held them out; no other target used it. The routing comments no longer call dep32 ungated. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
An RL weight sync (llmapi/rlhf_utils.py, and the NeMo-RL extension built on it) calls pre_reload_weights on every module, streams the new checkpoint in through partial loads, then runs process_weights_after_loading and post_load_weights. MegaMoECuteDslBase.pre_reload_weights refused every leaf, so an engine whose routed experts run on MEGAMOE_CUTEDSL raised NotImplementedError at its first weight sync. Each leaf now says whether it can take new weights in place (_hot_reload_refusal); the default refuses. The BF16 leaf opts in for a non-gated activation (Relu2, Nemotron-H's routed experts): its kernel reads w3_w1_weight / w2_weight themselves, with no derived buffers and no saved rebuild metadata, so partial loads copy each expert's FC1 / FC2 rows into place and the post-load hooks have nothing to rebuild. It then takes the standard MoE reload path, which still refuses any EPLB. UnquantizedMegaMoECuteDslMethod.load_weights accepts partial loads for it. The NVFP4 leaf (it releases its source weights after packing them) and a gated BF16 activation (the FC1 gate/up interleave needs both halves of an expert in one load) still refuse, each naming its reason. Tests: the refusal reasons (CPU), and the rlhf_utils sequence on an EP1 Relu2 layer: a full sync in two buckets, then a sync that carries only some experts' FC2, each against the fp32 reference, with the weight addresses unchanged and a CUDA graph captured before the first sync replaying on the new weights. On GB200: CPU + EP1 51 passed; the EP4 tests still pass. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…gines The RL weight-sync worker extensions (llmapi/rlhf_utils.py, set through ray_worker_extension_cls) stream partial checkpoints into the built model with ModelLoader.reload(..., allow_partial_loading=True). The nemotron_h targets load a whole checkpoint only: their load_weights takes no allow_partial_loading, so the loader asserts on the first bucket, and their weight tables assert that every key is present. An RL engine routed to a target therefore failed at its first weight sync, after training had started. route() now declines when ray_worker_extension_cls is set: modeling_v2='auto' keeps the built-in NemotronH model, which takes partial loads, and 'require' fails at build with the reason. AsyncLLM installs rlhf_utils.WorkerExtension by default, so the reason also says to pass ray_worker_extension_cls=None for an engine that never updates its weights. Routing tests cover the four BF16 layouts and the NVFP4 dep4 target, and a Ray engine without an extension keeping its target. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
test_cold_call_matches_the_reference runs the op in a child process and puts tests/unittest/_torch/modeling_v2 on the child's PYTHONPATH so it can import _validating. It took that root as Path(__file__).parents[1], which is models/nemotron_h/ops, so the child failed with "ModuleNotFoundError: No module named '_validating'". Derive the root from the __extra_import_path__ that the parent's own import of _validating uses. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…e KV cache compression Main now defaults modeling_v2 to "auto" and lets a target stay opt-in with @disable_v2_by_default. The nemotron_h targets were written as opt-in, so they take the decorator: with modeling_v2 unset the engine keeps the built-in NemotronH, and setting modeling_v2 still selects them (their GSM8K gates pass modeling_v2="require"). route() also declines kv_cache_compression_config: the targets were gated without KV cache compression and do not implement its compressed pages. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
… Ultra BF16 dep16/dep32 targets An RL weight sync (llmapi/rlhf_utils.py, and NeMo-RL's NcclExtension built on it) calls pre_reload_weights on every module, streams the new weights through bucketed partial loads (ModelLoader.reload(..., allow_partial_loading=True)), then runs the post-load hooks. The BF16 dep16 and dep32 targets now take it: - load_weights accepts allow_partial_loading. A bucket carries any subset of the checkpoint keys and is written into the same tensors, which captured CUDA graphs read. A key the target neither loads nor expects from a full update (another prefix, stacked experts) fails the bucket before anything is written. A bucket may convert dtypes, routed experts included; a whole checkpoint must still match. - post_load_weights builds the targets once; later calls refresh the bf16 copies of D and dt_bias in place and report, per rank, the checkpoint tensors the update left at their previous values. - route() admits RL engines (ray_worker_extension_cls) on dep16/dep32 with the CUTLASS (AUTO) and MegaMoE leaves, which take updates in place. TRTLLM's BF16 leaf re-stages every routed expert during an update and stays refused, as do the other targets; the cores refuse an update on TRTLLM as well. Validated on GB200 under the Ray orchestrator, wiping every weight and then restoring the checkpoint: GSM8K dep16 94.58 -> 0.0 -> 94.54, dep32 94.77 -> 0.0 -> 94.84, with the overlap scheduler on and off. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…t a scale-factor leg main's one-launch MegaMoE input staging (NVIDIA#21438) copies the activation's scale factors with its bytes, so the BF16 MegaMoE leaf, whose activation has none, failed on its first forward ("'NoneType' object has no attribute 'view'" in stage_megamoe_inputs, at engine warmup). The scale-factor leg is now compiled only when the call passes x_sf and sf_buffer, which must come together. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
… MoE to the native BF16 leaf main NVIDIA#21718 runs unquantized TRTLLM-Gen MoE on TRT-LLM's own BF16 cubins (trtllm.trtllm_gen.fused_moe.none) instead of FlashInfer's runner (flashinfer.trtllm_gen.fused_moe.none). The dep16, dep32 and tep8 targets still pinned the FlashInfer leaf, so moe_config.backend TRTLLM failed at model construction ("this target runs that request only on flashinfer.trtllm_gen.fused_moe.none"). The native leaf declares Relu2, top_k <= 32 and up to 2048 experts, with the same shuffled BlockMajorK layout. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ault tactic main NVIDIA#21438 gave the NVFP4 kind a quantized-combine default tactic in which every FC2 tile publishes its completion at once, so the token-back warps no longer wait for batches of four tiles. The BF16 kind, whose defaults are NVFP4's at the BF16 MMA tile K, still took the old non-bulk tactic for a quantized combine (test_megamoe_bf16_cpu_buffer_layout_and_tactics failed at the FC2 flag batch, (2, 4) vs (2, 1)). It now takes the same tactic; the bulk NVFP4-combine store stays NVFP4-only. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…t of the Nemotron-H targets The dep16, dep32 and tep8 targets import disable_v2_by_default (they stay opt-in under the "auto" modeling_v2 default), which runs at class definition and computes nothing, but test_nemotron_h_uncertified_generic_calls counted it as an undeclared stock import and failed for all three. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…act rows stage_megamoe_inputs (NVIDIA#21438) is compiled for compact operand rows (row stride equal to the width) and the TVM-FFI call rejects any other stride. run_moe compacts the routing tensors only when is_contiguous() is False, but a one-row (or empty) tensor reports contiguous whatever its row stride, e.g. one token's [1, top_k] slice of a wider argsort result. The BF16 EP4 test sends exactly that on its one-token rank and failed with "Mismatched idx_src.strides[0]". The helper now copies any source whose row stride is not its width; compact inputs, the normal case, pass through untouched. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…m a quant-kind entry The BF16 change appends the quant kind at position 18 of a MegaMoE problem key, for kinds other than nvfp4, and _default_megamoe_tactic_for_key read position 18 of any longer key as the kind. main's GLM-5.2 Rubin runner (NVIDIA#21585) passes its unique_id, an nvfp4 key with its own tag appended at position 18 ("glm-rubin-high-throughput"), so its default-tactic lookup took the tag for a quant kind and raised. Position 18 now counts only when it holds a known quant kind; anything else means an nvfp4 key. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…ck-scaled kinds main's one-round push dispatch (NVIDIA#21438) sizes and pushes the activation's scale-factor plane (_OneRoundRouter reads quant_kind.sf_vec_size), which the unscaled bf16 kind does not have; the kernel enabled it from the world size, the bucket and apply_topk_at_fc1 alone. A BF16 MegaMoE layer of a DeepSeek-V4 model at EP16 to EP64 would have failed to build. The bf16 kind keeps the size-broadcast dispatch. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…routing hooks main's FusedCommMoEScheduler (NVIDIA#21438) now asks the layer for its warmup placement route counters and whether a dynamic load balancer runs, and asks the backend whether it fuses route + quant. The stand-in layer and backend of test_fused_comm_chunks_survive_a_backend_that_reuses_its_output and test_fused_comm_single_chunk_hands_back_the_backend_output had none of them (AttributeError: '_warmup_placement_route_counters'). They now answer the way a layer without either feature and a backend without the fused launch do. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…zed experts With the unquantized MegaMoE CuteDSL leaf (trtllm.cutedsl.mega_moe.none), a mega_moe tactic on BF16 experts follows that family instead of falling back to AUTO, so test_a_tactic_serving_no_expert_format_builds_for_auto[dense] (NVIDIA#21778) failed with MEGAMOE_CUTEDSL != CUTLASS. Its dense case now uses FP8 block scales, a format no mega_moe implementation serves, and a new test pins the unquantized case to MEGAMOE_CUTEDSL. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…r the BF16 MegaMoE leaf The catalog NVIDIA#21778 generates from the implementation registry lacked trtllm.cutedsl.mega_moe.none (test_moe_impl_catalog failed as stale). Regenerated with scripts/generate_moe_impl_catalog.py's splice: the new leaf ranks 5, the implementations after it move down one, and two short aliases are no longer unique, so they lengthen: trtllm.mega_moe -> mega_moe.nvfp4 and cutedsl.none -> cutedsl.grouped_gemm.none. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
…hannel-major kernel main's NVIDIA#20916 rewrote the channel-major causal_conv1d kernel: it now takes the tanh form of SiLU above a cutoff for 16-bit outputs (the channel-last default keeps the exact form), and it reads x through its token stride (it had walked the token axis contiguously). The op and its test still assumed the old kernel: test_channel_last_and_channel_major_are_bitwise_identical and test_projection_slice_and_a_channel_major_copy_are_bitwise_identical failed on the activations, and the token_stride hazard case no longer misses the reference. The two kernels' published states still agree bitwise and their activations to bf16 rounding, which the renamed tests assert; the guard no longer refuses a token-strided channel-major x (a new test checks the op answers it), and keeps refusing a strided state. Signed-off-by: Guoming Zhang <137257613+nv-guomingz@users.noreply.github.com>
nv-guomingz
force-pushed
the
user/guomingz/mv2-nemotron-ultra-sm107
branch
from
November 16, 2026 16:39
306e3d9 to
a031280
Compare
Collaborator
|
Hi, heads-up: #22286 has merged, so ModelingV2 now lives at Since this PR touches modeling_v2, please rebase it onto the latest main. You can use these migration commands to move any files you added under the old directory and update any remaining references to the old path. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@coderabbitai summary
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.