Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
92db504
[None][feat] modeling_v2 qwen4_exp: family operations at the FP8 chec…
Wanli-Jiang Nov 10, 2026
fcc2c62
[None][feat] modeling_v2 qwen4_exp: Qwen3.8-Flash-Next-FP8 target for…
Wanli-Jiang Nov 10, 2026
597a862
[None][chore] modeling_v2 qwen4_exp: apply clang-format and cmake-for…
Wanli-Jiang Nov 10, 2026
4de86ae
[None][fix] modeling_v2 qwen4_exp: open the shared PLE table's rendez…
Wanli-Jiang Nov 10, 2026
6d3973b
[None][fix] modeling_v2 qwen4_exp: mirror wide in hc_up_gate_mean's r…
Wanli-Jiang Nov 10, 2026
8b84564
[None][fix] modeling_v2 qwen4_exp: allow one bf16 rounding in the HC …
Wanli-Jiang Nov 10, 2026
5837c73
[None][fix] modeling_v2 qwen4_exp: FP8 target kernel barriers, page-s…
Wanli-Jiang Nov 10, 2026
f2d0f00
[None][fix] modeling_v2 qwen4_exp: decline what the FP8 target does n…
Wanli-Jiang Nov 10, 2026
99c6bd2
[None][test] modeling_v2 qwen4_exp: list the FP8 target's accuracy te…
Wanli-Jiang Nov 11, 2026
8be84d8
[None][fix] modeling_v2 qwen4_exp: stock TRTLLM_FORCE_COMM_METHOD, sh…
Wanli-Jiang Nov 11, 2026
c8e1b12
[None][test] modeling_v2 qwen4_exp: FP8 target long-context and tp1 a…
Wanli-Jiang Nov 11, 2026
2c9de0a
[None][chore] modeling_v2 qwen4_exp fp8: drop the shared host PLE table
Wanli-Jiang Nov 11, 2026
049ea99
[None][feat] modeling_v2 qwen4_exp: PLE host table per rank, per serv…
Wanli-Jiang Nov 11, 2026
bd3462f
[None][fix] modeling_v2 qwen4_exp: precompile the tp2/tep2 HT replay …
Wanli-Jiang Nov 11, 2026
c581161
[None][test] modeling_v2 qwen4_exp: schedule multi-GPU stages when th…
Wanli-Jiang Nov 11, 2026
6a4ad97
[None][perf] modeling_v2 qwen4_exp: order the wide decode selection b…
Wanli-Jiang Nov 11, 2026
67295bc
[None][fix] modeling_v2 qwen4_exp: harden the FP8 MoE and hc all-redu…
Wanli-Jiang Nov 11, 2026
7dd2bcf
[None][fix] modeling_v2 qwen4_exp: direct-launch through the ops' pub…
Wanli-Jiang Nov 11, 2026
896e3e9
[None][test] modeling_v2 qwen4_exp: build the TPB test operands throu…
Wanli-Jiang Nov 11, 2026
9f45b45
[None][test] modeling_v2 qwen4_exp: pin the stable-JIT QSA kernels bi…
Wanli-Jiang Nov 11, 2026
fff2713
[None][chore] modeling_v2 qwen4_exp: build the window runner's keywor…
Wanli-Jiang Nov 11, 2026
a3ae84e
[None][fix] modeling_v2 qwen4_exp: decline in-forward snapshots the s…
Wanli-Jiang Nov 11, 2026
13c100b
[None][fix] modeling_v2 qwen4_exp: port the FP8 KV range probe to the…
Wanli-Jiang Nov 11, 2026
c4617a0
[None][fix] modeling_v2 qwen4_exp: run the vision tower on a chunk's …
Wanli-Jiang Nov 11, 2026
a70f1ca
[None][fix] modeling_v2 qwen4_exp: pass the draft-extend step its hos…
Wanli-Jiang Nov 11, 2026
1329367
[None][fix] modeling_v2 qwen4_exp: run eager draft steps' projections…
Wanli-Jiang Nov 11, 2026
739b873
[None][fix] modeling_v2 qwen4_exp: agree on the copy-engine exchange'…
Wanli-Jiang Nov 12, 2026
6e5c425
[None][fix] modeling_v2 qwen4_exp: refuse a foreign-uid PLE broker ow…
Wanli-Jiang Nov 12, 2026
066001d
[None][fix] modeling_v2 qwen4_exp: decline a sliding window at an inf…
Wanli-Jiang Nov 12, 2026
055596b
[None][fix] modeling_v2 qwen4_exp: score IndexShare draft context row…
Wanli-Jiang Nov 12, 2026
3936b67
[None][perf] modeling_v2 qwen4_exp: order a mixed step's generation r…
Wanli-Jiang Nov 12, 2026
35d3bac
[None][fix] modeling_v2 qwen4_exp: run the vision tower once per fram…
Wanli-Jiang Nov 12, 2026
19ae582
[None][fix] modeling_v2 qwen4_exp: honor the engine's all-reduce stra…
Wanli-Jiang Nov 12, 2026
1c0db6b
[None][fix] modeling_v2 qwen4_exp: harden the node PLE host table
Wanli-Jiang Nov 12, 2026
5dd2270
[None][fix] modeling_v2 qwen4_exp: acquire at cluster scope on the FP…
Wanli-Jiang Nov 12, 2026
48e7339
[None][fix] modeling_v2 qwen4_exp: run the FP8 low-M projections and …
Wanli-Jiang Nov 12, 2026
fd37f37
[None][fix] modeling_v2 qwen4_exp: compile every split-K decode plan …
Wanli-Jiang Nov 12, 2026
c377414
[None][fix] modeling_v2 qwen4_exp: precompile the norm-folded GDN rep…
Wanli-Jiang Nov 12, 2026
20945fe
[None][chore] modeling_v2 qwen4_exp: state the measured GDN replay co…
Wanli-Jiang Nov 12, 2026
ffeeb68
[None][fix] modeling_v2 qwen4_exp: address the FP8 target's GDN repla…
Wanli-Jiang Nov 12, 2026
dc7f172
[None][fix] modeling_v2 qwen4_exp: make the direct launches local cop…
Wanli-Jiang Nov 13, 2026
583d116
[None][fix] modeling_v2 qwen4_exp: read the AutoTuner capture state w…
Wanli-Jiang Nov 13, 2026
80aea32
[None][fix] modeling_v2 qwen4_exp: call the raw all-reduce at AUTO fo…
Wanli-Jiang Nov 13, 2026
c4cf317
[None][fix] modeling_v2 qwen4_exp: document the multi-GPU trigger for…
Wanli-Jiang Nov 13, 2026
4b31a21
[None][fix] modeling_v2 qwen4_exp: keep the HC inject-norm Lamport pr…
Wanli-Jiang Nov 13, 2026
532864b
[None][test] modeling_v2 qwen4_exp: fit the FP8 target's post-merge t…
Wanli-Jiang Nov 16, 2026
29aca83
[None][test] modeling_v2 qwen4_exp: drop the FP8 target's MNNVL tp_su…
Wanli-Jiang Nov 16, 2026
3723256
[None][fix] modeling_v2 qwen4_exp: follow main's sigmoid-mul and gate…
Wanli-Jiang Nov 16, 2026
30c927a
[None][test] modeling_v2 qwen4_exp: restore the FP8 target's MNNVL tp…
Wanli-Jiang Nov 16, 2026
5969704
[None][chore] modeling_v2 qwen4_exp: re-sort imports and fix a packag…
Wanli-Jiang Nov 17, 2026
2fa681c
[None][fix] modeling_v2 qwen4_exp: join the PLE broker's serve thread…
Wanli-Jiang Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
175 changes: 175 additions & 0 deletions tensorrt_llm/_torch/modeling_v2/models/qwen4_exp/chunk_broker.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""Hand one process's exportable allocation to the other processes of the host that ask for it.

A segment is named by a key. The first process to bind the key's abstract Unix socket
(`\\0trtllm-ple-<uid>-<key>`) owns it: it creates and fills the allocation, then serves the
allocation's file descriptor to every process that connects (`SCM_RIGHTS`). Abstract names carry
no permissions, so both sides check the other's uid through `SO_PEERCRED`: the owner refuses a
peer of another uid, and a waiter whose name is held by another uid's socket gets
`PermissionError` at once. A process that cannot bind connects and waits, at most `timeout_s`
(also when the name is bound but nothing listens on it). When the owner exits or aborts before
publishing, the kernel closes its sockets and the waiter tries to own the segment itself.
Nothing is written to the filesystem, so a crash leaves nothing behind.

Abstract socket names belong to a network namespace: processes in different namespaces do not
meet, and each keeps its own copy.
"""

from __future__ import annotations

import errno
import json
import os
import socket
import struct
import threading
import time
from typing import Callable, Optional

OWNER = "owner"
ATTACHED = "attached"

_CRED = struct.Struct("iII") # struct ucred: pid_t pid, uid_t uid, gid_t gid
_POLL_SECONDS = 1.0


def socket_name(key: str) -> str:
"""The abstract socket name of `key` (per uid)."""
return f"\0trtllm-ple-{os.getuid()}-{key}"


def _peer_uid(conn: socket.socket) -> int:
"""The uid of the process at the other end of the connected Unix socket `conn`."""
_, uid, _ = _CRED.unpack(conn.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, _CRED.size))
return uid


class OwnedSegment:
"""The owner's side: serves `publish`ed `(fd, meta)` to every same-uid peer, until `close`."""

def __init__(self, server: socket.socket, key: str) -> None:
self.key = key
self._server = server
self._ready = threading.Event()
self._closed = threading.Event()
self._fd = -1
self._meta: dict = {}
self._answers: list[threading.Thread] = []
self._thread = threading.Thread(
target=self._serve, name=f"ple-broker-{key[:12]}", daemon=True
)
self._thread.start()

def publish(self, fd: int, meta: dict) -> None:
"""Make the allocation available. `fd` stays owned by the caller and must stay open."""
self._fd = fd
self._meta = dict(meta)
self._ready.set()

def close(self) -> None:
"""Stop serving; peers still waiting see the connection close and retry."""
self._closed.set()
try:
# Closing alone does not wake the thread blocked in `accept`; shutting down does.
self._server.shutdown(socket.SHUT_RDWR)
except OSError:
pass
self._server.close()
# `_serve` alone starts answers: once it has returned, the list of answers is final.
self._thread.join(_POLL_SECONDS * 5)
for thread in self._answers:
thread.join(_POLL_SECONDS * 5)

def _serve(self) -> None:
while not self._closed.is_set():
try:
conn, _ = self._server.accept()
except OSError:
# `close`, or `accept` failing (EMFILE, ENFILE, ENOMEM, ...). Either way the name
# is freed with the thread: connected peers already answered keep their answer,
# and every other peer sees its connection reset and owns the key itself.
self._server.close()
return
answer = threading.Thread(target=self._answer, args=(conn,), daemon=True)
self._answers = [t for t in self._answers if t.is_alive()] + [answer]
answer.start()

def _answer(self, conn: socket.socket) -> None:
with conn:
try:
if _peer_uid(conn) != os.getuid():
return
while not self._ready.wait(_POLL_SECONDS):
if self._closed.is_set():
return
if self._closed.is_set():
return
socket.send_fds(conn, [json.dumps(self._meta).encode()], [self._fd])
except OSError:
return


def try_own(key: str) -> Optional[OwnedSegment]:
"""Own `key` if no process does (non-blocking), else None."""
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM | socket.SOCK_CLOEXEC)
try:
server.bind(socket_name(key))
server.listen(64)
except OSError as error:
server.close()
if error.errno == errno.EADDRINUSE:
return None
raise
return OwnedSegment(server, key)


def acquire(
key: str, timeout_s: float, log: Callable[[str], None] = print, report_s: float = 60.0
) -> tuple:
"""`(OWNER, OwnedSegment)` when no process owns `key`, else `(ATTACHED, fd, meta)` once
the owner publishes; an owner that exits, aborts or stops accepting first hands `key` to its
waiters. Raises `TimeoutError` when no owner has published within `timeout_s`, and
`PermissionError` when a process of another uid holds the name."""
name = socket_name(key)
deadline = time.monotonic() + timeout_s
next_report = time.monotonic() + report_s
while True:
owned = try_own(key)
if owned is not None:
return OWNER, owned
conn = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM | socket.SOCK_CLOEXEC)
try:
conn.connect(name)
except (ConnectionRefusedError, FileNotFoundError):
# The owner just exited, or is between `bind` and `listen` (or never listens).
conn.close()
if time.monotonic() >= deadline:
raise TimeoutError(f"no owner of {key!r} listens within {timeout_s:.0f} s")
time.sleep(0.05)
continue
got: Optional[tuple] = None
try:
owner_uid = _peer_uid(conn)
if owner_uid != os.getuid():
raise PermissionError(f"the owner of {key!r} runs as uid {owner_uid}")
while got is None:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise TimeoutError(f"no allocation for {key!r} within {timeout_s:.0f} s")
conn.settimeout(min(remaining, report_s))
try:
got = socket.recv_fds(conn, 4096, 1)
except socket.timeout:
if time.monotonic() >= next_report:
log(f"ple host table: waiting for the owner of {key!r} to publish it")
next_report = time.monotonic() + report_s
except ConnectionResetError:
# The owner closed its listener before accepting this connection.
got = (b"", [], 0, None)
finally:
conn.close()
msg, fds, _, _ = got
if fds:
return ATTACHED, fds[0], json.loads(msg.decode())
# The connection closed without a descriptor: the owner exited or aborted. Retry.
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES.
# All rights reserved. SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not
# use this file except in compliance with the License. You may obtain a copy of
# the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations under
# the License.
#

# qwen4_exp native code. Each target keeps its own in <target>/csrc/; see the
# modeling_v2 README.
add_modeling_v2_family(qwen4_exp)

# The FP8 target routes on SM103 only (its kernels need only SM90+ features).
# The hyper-connection all-reduce uses the stock all-reduce's IPC workspace,
# which exists only in multi-device builds.
add_modeling_v2_target(
qwen4_exp qwen3_8_flash_next_fp8__sm_103__unified
ARCHS 103a
SOURCES *.cu *.cpp
REQUIRES MULTI_DEVICE)
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,63 @@

The gated attention's output gate, between the attention core and o_proj, in
one Triton kernel (`torch.ops.trtllm.fused_sigmoid_mul_inplace`). Called by the
qwen4_exp target (`qwen3_8_flash_next_nvfp4__sm_103__default`) at every layout -- its
trunk, gate off the fused input-projection row, and its MTP layer, gate as the strided
half of the q projection.
qwen4_exp targets at every layout -- the trunk, gate off the fused input-projection row,
and the MTP layer, gate as the strided half of the q projection.
"""

from __future__ import annotations

import torch
import triton

# Registered by the module that defines it: after `import
# tensorrt_llm._torch.custom_ops` alone, torch.ops.trtllm has no
# `fused_sigmoid_mul_inplace` (measured).
import tensorrt_llm._torch.attention.kernels.fused_qk_norm_rope_gate # noqa: F401
import tensorrt_llm._torch.attention.kernels.fused_qk_norm_rope_gate as _gate_mod
from tensorrt_llm._torch.modeling_v2.catalog._op import Arch, Cell, OpWrapper


def _launch(attention_output: torch.Tensor, gate: torch.Tensor) -> bool:
"""The registered op's launch, made here when its preconditions hold: the same kernel,
grid, block, warps and PDL policy, without the custom-op dispatch around it. False
(nothing launched) when they do not, for the op to refuse loudly."""
if attention_output.dim() != 2 or attention_output.stride(-1) != 1 or gate.stride(-1) != 1:
return False
num_tokens, hidden = attention_output.shape
if gate.dim() == 3:
if gate.shape[0] != num_tokens or gate.shape[1] * gate.shape[2] != hidden:
return False
head_dim, gate_stride_head = gate.shape[2], gate.stride(1)
elif gate.dim() == 2 and gate.shape == attention_output.shape:
head_dim, gate_stride_head = hidden, hidden
else:
return False
if num_tokens == 0:
return True
block = min(triton.next_power_of_2(hidden), 1024 if num_tokens < 1024 else 2048)
pdl = (
dict(LAUNCH_WITH_PDL=True, launch_pdl=True)
if _gate_mod._sigmoid_mul_pdl(attention_output.device.index)
else {}
)
_gate_mod._fused_sigmoid_mul_kernel[(num_tokens, triton.cdiv(hidden, block))](
attention_output,
attention_output,
gate,
attention_output, # scale_ptr, read only for an FP8 output
attention_output.stride(0),
attention_output.stride(0),
gate.stride(0),
gate_stride_head,
hidden_size=hidden,
head_dim=head_dim,
block_size=block,
num_warps=4,
**pdl,
)
return True


#: The checkpoint's gated attention: 24 q heads of 256, q projection [T, 24, 512].
_CKPT = dict(heads=24, head_dim=256, dtype=torch.bfloat16, gate="interleaved")

Expand Down Expand Up @@ -135,7 +177,12 @@ class FusedSigmoidMulInplace(OpWrapper):
the running device at the first call of each (hidden, head_dim, block)
specialization, no autotuner: grid `(T, cdiv(hidden, block))`,
`block = min(next_pow2(hidden), 1024 if T < 1024 else 2048)`, 4 warps. The
cells ran on sm_103.
cells ran on sm_103. With `direct_launch=True` (the FP8 target), `_launch` makes
that same launch itself where the preconditions below hold and skips only the
custom-op dispatch (15 instead of 28 us of host time a call on GB300);
otherwise the op is called and refuses the call. `_launch` is a copy of the
op's launch: the test holds it bitwise equal to the op at the direct-launch
cells and more shapes, so a change to the op's launch fails there.

Call sites, each the attention output gate between the attention core and
o_proj:
Expand Down Expand Up @@ -192,7 +239,10 @@ class FusedSigmoidMulInplace(OpWrapper):
gate) at 24 heads (`[T, 24, 512]`, hidden 6144) at 1 token, and at 6 heads
(a compact `[T, 6, 512]`, hidden 1536) at 1, 128, 512, 1024 and 8192 rows;
CUDA-graph capture and replay of the 1536-wide fused-row form at 8 rows and
of the 6-head 3-D gate at 1, 128 and 1024 rows.
of the 6-head 3-D gate at 1, 128 and 1024 rows. With `direct_launch` (the FP8
target's cells in the test): the 2-D window at hidden 3072 (row stride 7424,
gate column 3584) at 1 and 8192 rows, the 24-head 3-D gate at 128 and 8192
rows, and graph capture of the 1536-wide fused-row form.

Hard limits
T * row stride token offset computed in int64 boundary 2^63 not probed
Expand All @@ -202,19 +252,24 @@ class FusedSigmoidMulInplace(OpWrapper):

Not certified: 0 tokens; rows past 8192; the 3-D gate at a token stride
other than its own q_proj's (heads x 2 x head_dim, e.g. a gate inside a
fused q|k|v row); the 24-head strided q_proj half above 1 token (a dep4
rank's MTP layer: drafter 1..32 rows and its verify); a strided
`attention_output`; a gate dtype other than `attention_output.dtype`;
fp16/fp32 operands; hidden sizes other than 1536 and 6144; CUDA-graph
capture of the 6144-wide forms; torch.compile.
fused q|k|v row); the 24-head strided q_proj half above 1 token without
`direct_launch`; a strided `attention_output`; a gate dtype other than
`attention_output.dtype`; fp16/fp32 operands; hidden sizes other than 1536,
3072 and 6144; CUDA-graph capture of the 6144-wide forms; torch.compile.
"""

def __call__(self, attention_output: torch.Tensor, gate: torch.Tensor) -> torch.Tensor:
torch.ops.trtllm.fused_sigmoid_mul_inplace(attention_output, gate)
def __call__(
self, attention_output: torch.Tensor, gate: torch.Tensor, direct_launch: bool = False
) -> torch.Tensor:
if not (direct_launch and _launch(attention_output, gate)):
torch.ops.trtllm.fused_sigmoid_mul_inplace(attention_output, gate)
return attention_output

def reference(self, attention_output: torch.Tensor, gate: torch.Tensor) -> torch.Tensor:
def reference(
self, attention_output: torch.Tensor, gate: torch.Tensor, direct_launch: bool = False
) -> torch.Tensor:
"""What `attention_output` holds after the call; neither input is mutated.
`direct_launch` does not change it.

`gate.reshape(T, hidden)` is the per-head `(t, i // D, i % D)` read, for
the 2-D and 3-D forms alike.
Expand Down
Loading
Loading