Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2ccbda8
[None][feat] MiniMax-M3: load the compressed-tensors NVFP4 MoE packaging
pranav-nvidia Oct 23, 2026
f4e8e70
[None][perf] MiniMax-M3: BF16-split tensor-core router GEMM
pranav-nvidia Oct 23, 2026
3bc6fb1
[None][perf] MiniMax-M3 sparse attention: trim per-step host work
pranav-nvidia Oct 23, 2026
d63fe42
[None][feat] Allow NVFP4 KV cache for MiniMax-M3 sparse attention on …
pranav-nvidia Oct 23, 2026
186d513
[None][fix] MiniMax-M3 VL: use the serve-rendered prompt and count vi…
pranav-nvidia Oct 23, 2026
50cee82
[None][fix] MiniMax-M3 VL: keep speculative decoding and KV reuse wor…
pranav-nvidia Oct 23, 2026
6975733
[None][fix] Take the token-ID fast path for MiniMax-M3 VL image requests
pranav-nvidia Oct 23, 2026
02f75fc
[None][feat] Serve MiniMax-M3 with a standalone DSpark drafter
pranav-nvidia Oct 23, 2026
493ec2c
[None][fix] Size the standalone drafter's KV mirror for disaggregated…
pranav-nvidia Oct 23, 2026
eae7bb8
[None][fix] Keep the DFlash block write on an owned page after an una…
pranav-nvidia Oct 23, 2026
29880cc
[None][fix] Keep MiniMax-M3 streaming and non-streaming reasoning par…
pranav-nvidia Oct 23, 2026
0280b23
[None][doc] Document MiniMax-M3.1-Flash serving in the MiniMax-M3 guide
pranav-nvidia Oct 23, 2026
7193d64
[None][fix] Serve MiniMax-M3 chat with the tokenizer's template and t…
pranav-nvidia Oct 23, 2026
59256ca
[None][chore] Set MiniMax-M3.1-Flash side-branch release version to 1…
pranav-nvidia Oct 23, 2026
5d72b49
[None][fix] Enforce container wheel version and source commit (#20564)
chzblych Oct 20, 2026
47ecfa7
[None][chore] Set MiniMax-M3.1-Flash side-branch version to 1.4.0a0
chzblych Oct 23, 2026
98660c5
[None][fix] disable bolt (#20496)
tburt-nv Oct 17, 2026
40ca75b
[None][perf] Add GB300 MiniMax-M3.1 low-latency decode targets
peihu-nv Oct 23, 2026
647ee5c
[None][fix] Share per-row RNG subsequence constant with chain sampling
peihu-nv Oct 24, 2026
1113060
[None][test] Restore GB300 TP4 test operand helpers
peihu-nv Oct 24, 2026
9d97565
[None][fix] Restore DecoderRunner optional state initialization
peihu-nv Oct 25, 2026
162e93e
[None][feat] Add Rubin (sm_107) MiniMax-M3.1 low-latency decode targets
peihu-nv Oct 26, 2026
bc6d981
[None][fix] Disaggregated generation: one-hot draft probs on the firs…
peihu-nv Oct 26, 2026
fd38234
[None][fix] Gate TOPP_V6 to sm_107 and keep the token-array copy to h…
peihu-nv Oct 26, 2026
0f4ab31
[None][fix] Arm the shared MiniMax-M3.1 sm_107 paths only for the tar…
peihu-nv Oct 26, 2026
0c88f87
[None][test] Expect a device token array to be rejected by LlmRequest
peihu-nv Oct 26, 2026
3f48ba4
[None][test] Cover the KV prefetch startup state when the model does …
peihu-nv Oct 26, 2026
6aaa96e
[None][fix] Arm the shared MiniMax-M3.1 sm_107 opt-ins at routing
peihu-nv Oct 26, 2026
bb04179
[None][fix] Avoid batch-shape specialization in DSpark drafter inputs
peihu-nv Oct 27, 2026
b3f097d
[None][test] Cover DSpark input strides and context-view specializations
peihu-nv Oct 27, 2026
6355f11
[None][fix] Await the iteration-stats RPC instead of blocking the eve…
peihu-nv Oct 27, 2026
75dfb89
[None][feat] Add a phase-armed Nsight window and a per-rank host-path…
peihu-nv Oct 27, 2026
a3f86ac
[None][test] Run the iteration-stats RPC test in the CPU test list
peihu-nv Oct 27, 2026
6cb186f
[None][perf] Run the MiniMax-M3.1 TP2 router as an exact half-row TF3…
peihu-nv Oct 27, 2026
41cd32f
[None][test] Cover the MiniMax-M3.1 TP2 router half-row pin
peihu-nv Oct 27, 2026
82c5e77
[None][perf] Time the executor with GPU timestamps on the MiniMax-M3.…
peihu-nv Oct 27, 2026
f0af3b6
[None][fix] Drop the Triton stand-in stamp from the executor's GPU ti…
peihu-nv Oct 27, 2026
8b1d98e
[None][chore] Drop the unused separate-library loader from the TP2 ro…
peihu-nv Oct 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
459 changes: 448 additions & 11 deletions 3rdparty/vendor_patches/flashinfer-prims-ts.patch

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions 3rdparty/vendor_sources.lock.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ vendors:
include:
- '**/*.py'
patch: 3rdparty/vendor_patches/flashinfer-prims-ts.patch
patch_digest: sha256:65cc08e20a9cc7bdc190eb6f3e79124986dced74394c1f3250ae2d100a8ee340
digest: sha256-tree-v1:e411b6c6679474d362b166e586df4276f52322d75a35c59c782d6266143e560e
patch_digest: sha256:e1f3f573c2723e432a8f4495c0f5329099b54f00856779996ac9b7f4ff0c21f7
digest: sha256-tree-v1:b7be1c7ddbbcbd51dac753bdd9b0dd5cbeb5b122330a5c8d1366b79e7e635cba
openengine-proto:
url: https://github.com/ai-dynamo/openengine.git
tag: v0.1.0
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ TensorRT LLM
[![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31012/)
[![cuda](https://img.shields.io/badge/cuda-13.4.1-green)](https://developer.nvidia.com/cuda-downloads)
[![torch](https://img.shields.io/badge/torch-2.14.0-green)](https://pytorch.org)
[![version](https://img.shields.io/badge/release-1.4.0rc1-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py)
[![version](https://img.shields.io/badge/release-1.4.0a0-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py)
[![license](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE)

[Architecture](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)   |   [Performance](https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html)   |   [Examples](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)   |   [Documentation](https://nvidia.github.io/TensorRT-LLM/)   |   [Roadmap](https://github.com/NVIDIA/TensorRT-LLM/issues?q=is%3Aissue%20state%3Aopen%20label%3Aroadmap)
Expand Down
59 changes: 58 additions & 1 deletion cpp/tensorrt_llm/kernels/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,13 @@ list(FILTER SRC_CU EXCLUDE REGEX "fusedSampling/.*")
list(FILTER SRC_CPP EXCLUDE REGEX "marlin/.*")
list(FILTER SRC_CU EXCLUDE REGEX "marlin/.*")

# Scope tcgen05 attention sources to their supported architectures.
list(FILTER SRC_CU EXCLUDE REGEX
"minimaxM3/(fusedDecodeAttn|attnMega|attnC1)\\.cu$")
# The MiniMax-M3 sm_107 decode kernels are built as their own sm_107-only OBJECT
# library below.
list(FILTER SRC_CU EXCLUDE REGEX "minimaxM3Sm107Tp[248]/.*")

if(NOT ENABLE_MULTI_DEVICE)
list(FILTER SRC_CU EXCLUDE REGEX "customAllReduceKernels*.*cu$")
endif()
Expand Down Expand Up @@ -111,10 +118,60 @@ if(MARLIN_SRC)
set_cuda_architectures(marlin_src 89 90 120f)
endif()

# Preserve the FlashInfer-matched compilation for sampling only. Do not enable
# fast math globally: attention exactness depends on its flags.
set_source_files_properties(
${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetSm103.cu
${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/toppRenormDetV3Sm103.cu
${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/softmaxMcSm103.cu
${CMAKE_CURRENT_SOURCE_DIR}/minimaxM31/softmaxPathBSm107.cu
PROPERTIES COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CUDA>:--use_fast_math>")

# Preserve the historical architecture-scoped attention object library.
add_library(
minimax_m3_attn_src OBJECT minimaxM3/fusedDecodeAttn.cu minimaxM3/attnMega.cu
minimaxM3/attnC1.cu)
set_property(TARGET minimax_m3_attn_src PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET minimax_m3_attn_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
set_cuda_architectures(minimax_m3_attn_src 100 103)

# MiniMax-M3 sm_107 decode kernels (minimaxM3Sm107Tp{2,4,8}/): written for
# sm_107 only (PDL griddepcontrol, sm_10x-only PTX), so they are compiled for
# the sm_10x entries of this build's architectures only. A build without an
# sm_10x architecture compiles them for sm_107a: their torch ops link against
# the host launchers, and only the sm_107 targets launch them.
file(GLOB MINIMAX_M3_SM107_SRC minimaxM3Sm107Tp2/*.cu minimaxM3Sm107Tp4/*.cu
minimaxM3Sm107Tp8/*.cu)
add_library(minimax_m3_sm107_src OBJECT ${MINIMAX_M3_SM107_SRC})
set_property(TARGET minimax_m3_sm107_src PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET minimax_m3_sm107_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS
ON)
target_include_directories(
minimax_m3_sm107_src
PRIVATE
$<TARGET_PROPERTY:${INTERNAL_CUTLASS_KERNELS_TARGET},INTERFACE_INCLUDE_DIRECTORIES>
)
target_link_libraries(minimax_m3_sm107_src PRIVATE trtllm_gen_fmha_interface)
set(MINIMAX_M3_SM107_ARCHS ${CMAKE_CUDA_ARCHITECTURES})
list(FILTER MINIMAX_M3_SM107_ARCHS INCLUDE REGEX "^10[0-9]")
if(NOT MINIMAX_M3_SM107_ARCHS)
set(MINIMAX_M3_SM107_ARCHS 107a-real)
endif()
# SM107 SASS next to the family build: a family (100f) cubin is scheduled for
# sm_100 and SM107 runs it as is; the LC2 / LC2-split kernels built for sm_107
# are 3.6-3.9% faster, bit for bit. The driver prefers the exact-arch cubin.
if(NOT "107a-real" IN_LIST MINIMAX_M3_SM107_ARCHS AND NOT "107f-real" IN_LIST
MINIMAX_M3_SM107_ARCHS)
list(APPEND MINIMAX_M3_SM107_ARCHS 107f-real)
endif()
set_property(TARGET minimax_m3_sm107_src PROPERTY CUDA_ARCHITECTURES
${MINIMAX_M3_SM107_ARCHS})

add_library(
kernels_src STATIC
${SRC_CPP} ${SRC_CU} ${MOE_KERNELS_SRC}
$<$<TARGET_EXISTS:marlin_src>:$<TARGET_OBJECTS:marlin_src>>)
$<$<TARGET_EXISTS:marlin_src>:$<TARGET_OBJECTS:marlin_src>>
$<TARGET_OBJECTS:minimax_m3_attn_src> $<TARGET_OBJECTS:minimax_m3_sm107_src>)
set_property(TARGET kernels_src PROPERTY POSITION_INDEPENDENT_CODE ON)
set_property(TARGET kernels_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON)
target_include_directories(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@

namespace tensorrt_llm
{
//! Philox subsequences from here up hold per-row streams. A shared seed/offset makes the row
//! index the subsequence, which never reaches them.
constexpr uint64_t kPerRowSubsequenceBase = 1ull << 63;

namespace kernels
{

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,6 @@ struct RowParams
bool needMinP;
};

//! Philox subsequences from here up hold per-row streams. A shared seed/offset makes the row
//! index the subsequence, which never reaches them.
constexpr uint64_t kPerRowSubsequenceBase = 1ull << 63;

//! Seeds one row's Philox stream (see FusedSamplingParams::perRowRng).
__device__ inline void initRowRng(FusedSamplingParams const& p, int row, curandStatePhilox4_32_10_t* state)
{
Expand Down
Loading