Skip to content

[TRTLLM-17451][feat] Add a global memory monitor - #21113

Merged
zhaoyangwang-nvidia merged 5 commits into
NVIDIA:mainfrom
zhaoyangwang-nvidia:feat/memory-ledger
Nov 21, 2026
Merged

zhaoyangwang-nvidia merged 5 commits into
NVIDIA:mainfrom
zhaoyangwang-nvidia:feat/memory-ledger

Conversation

@zhaoyangwang-nvidia

@zhaoyangwang-nvidia zhaoyangwang-nvidia commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

JIRA: TRTLLM-17451

Description

Why

When a rank runs out of memory today, all it reports is the out-of-memory error. It does not say which memory filled the device, what changed since startup, or what to adjust. The pieces that could answer this are spread over several allocators, and they do not add up naively. Measured on B200:

  • MemoryCounters overlaps torch_reserved rather than adding to it. A 512 MiB allocation raises both by 512 MiB, so summing the two doubles the figure.
  • PagePool.num_pages() reports configured capacity, not occupancy. It still reported 128 MiB after the driver had taken all of it back.
  • MNNVL memory is backed before it reaches MAPPED. A rollup that counts only MAPPED reports zero during an OOM.
  • A CUDA graph pool's free space is not reusable. Counting it as free cache overstates headroom.

Why one PR

The report is only useful if its rows add up to what the process holds: a row that cannot be trusted to be complete sends the reader to the wrong component, which is worse than no report. That makes the data sources one unit -- the PyTorch pools, the KV cache managers, the C++ ledger, the communication buffers and the startup steps each close a gap the others leave, and leaving one out turns its memory into unexplained overhead. The on-demand API, the MPI gathering and the serve route are thin readers of the same report. The C++ side is a small ledger plus record calls at the sites that allocate outside PyTorch. The docs page describes the whole.

What it does

The monitor is configured with the prototype memory_monitor_config in the LLM arguments (mode: oom by default, full or off; ranks, record_stacks, periodic_report_interval_s, oom_report_interval_s, report_dir). Each field has a TLLM_MEM_MONITOR* environment variable that overrides it, so a run can be diagnosed without changing its configuration; the variables are folded into the arguments once, where they are built, and reach every rank inside memory_monitor_config. The default oom mode reports an out-of-memory error that reaches the executor's own error handling (a failed request, a crashed executor loop, a failed wakeup) and installs nothing process-wide; full also records startup, installs process-wide hooks so errors on background threads and logged errors are reported too, reports once startup has completed (warmup included) and compares later reports with that point; off produces no reports on its own. The monitor follows one executor per process, the first one alive; another executor in the same process runs without it, with one warning. Out-of-memory reports are logged at ERROR, so they are visible at the default log level. The summary comes first and is written so a reader does not need to know which allocator holds what.

Mode memory_monitor_config.mode / TLLM_MEM_MONITOR Reports produced Startup cost Serving cost
Out-of-memory (default) oom / unset One report per rank when an OOM reaches the executor's error handling (failed request, crashed executor loop, failed wakeup; VisualGen: failed request), logged at ERROR. Every group and the total; no startup breakdown, no since warmup. LLM.get_memory_report() works; under MPI only rank 0 answers. Pools classified once when the executor has started None until an OOM
Full full / 1 Also a report when startup completes (per-step breakdown, warmup baseline); later reports show since warmup, the startup plan and the overhead split by when it appeared. Process-wide hooks (logger ERROR filter, threading.excepthook, torch OOM observer). Optional periodic reports and stack recording; every rank gathered under MPI; GET /memory_report with a runtime-control key and a single frontend. Allocation history during startup only; measured +0-1.5 % startup, +0.1-0.6 GiB host RSS per rank at 70B/DeepSeek-V3 None per iteration; a report costs milliseconds
Off off / 0 Nothing on its own; LLM.get_memory_report() still answers on request. Pools classified once when the executor has started None

Every allocator and every component that owns GPU memory feeds one monitor, which places each byte in exactly one group named for its purpose:

flowchart LR
    subgraph SRC["Allocators and components"]
        direction TB
        S1["PyTorch caching allocator<br/>one snapshot per report, split by pool:<br/>default, CUDA graph, MemPool"]
        S2["KV cache managers<br/>registered when built, held weakly<br/>v2 StorageStatistics, v1 allocated bytes"]
        S3["Loaded model<br/>claimed weakly; live parameters and<br/>buffers matched to allocator blocks"]
        S4["Startup steps and regions (full mode)<br/>allocation tags per executor step;<br/>NVML deltas for LoRA,<br/>KV transfer, torch.distributed NCCL"]
        S5["C++ native ledger<br/>recorded at alloc/free: NCCL comms,<br/>NVLS GEMM+AR, MoE profiler, Ulysses,<br/>EPLB, userbuffers"]
        S6["NCCL window allocator<br/>pooled window buffers"]
        S7["Component registries<br/>IPC buffers (custom AR, TP exchange),<br/>MNNVL, multicast AR, SymmMem, DeepEP,<br/>DWDP pages and local weight shards"]
        S8["NVML<br/>this process's total, by GPU UUID"]
    end

    M(["Memory monitor<br/>each byte in exactly one group;<br/>overhead = NVML total minus all others,<br/>so the groups sum to the process total"])

    subgraph OUT["Report groups"]
        direction TB
        G1["KV cache<br/><i>pool capacity per pool group (main, draft, cross),<br/>KV auxiliary tensors; free blocks included;<br/>device-backed pools only, host-backed listed apart</i>"]
        G2["Model weights<br/><i>parameters and buffers, DWDP pages and<br/>local shards, EPLB expert slots</i>"]
        G3["Runtime buffers<br/><i>tagged startup allocations: model, sampler,<br/>spec decoding, executor; C++ workspaces</i>"]
        G4["CUDA graphs<br/><i>graph private pools, live and free;<br/>free space only the graph can reuse</i>"]
        G5["Communication buffers<br/><i>NCCL comms and windows, IPC buffers, MNNVL,<br/>NVLS/multicast, SymmMem, DeepEP, Ulysses,<br/>userbuffers</i>"]
        G6["PyTorch, unattributed<br/><i>live with no known owner: activations, leaks;<br/>split by allocating code with record_stacks</i>"]
        G7["PyTorch cache (free)<br/><i>reserved by PyTorch, not in use, reusable</i>"]
        G8["CUDA and library overhead<br/><i>CUDA context, cuBLAS, NCCL internals, loaded code,<br/>anything unmeasured; split by when it appeared</i>"]
    end

    S1 & S2 & S3 & S4 & S5 & S6 & S7 & S8 --> M
    M --> G1 & G2 & G3 & G4 & G5 & G6 & G7 & G8
Loading

A real out-of-memory report (memory_monitor_config=MemoryMonitorConfig(mode="full"), no environment variables; Llama-3.1-8B-Instruct, TP=2 on B200, rank 1). After a normal request, a logits processor leaked 4 GiB per step on every rank and then asked for more than was free. Both ranks reported; the request failed with RequestError and serving continued with unchanged outputs:

=== TensorRT-LLM GPU memory monitor: out_of_memory ===
  pid=858 rank=1 device=1
  OUT OF MEMORY: PyTorch could not allocate 73.41 GiB on device 1 (device free 72.13 GiB; PyTorch may use up to 91.93 GiB; PyTorch held 19.74 GiB live, 19.77 GiB reserved). Error: [TRT-LLM] [E] [_torch][RANK 1] Encountered an error in forward function: CUDA out of memory. Tried to allocate 73.41 GiB. GPU 1 has a total capacity of 178.39 GiB of which 72.13 GiB is free. Including
  This process uses 106.08 GiB of 178.39 GiB (other processes 0.18 GiB, free 72.13 GiB).

                                      size   share   since warmup
  KV cache                       83.03 GiB   78.3%      +0.00 GiB   allocated up front; free blocks stay resident
  PyTorch, unattributed          12.00 GiB   11.3%     +12.00 GiB   in use by PyTorch, owner not identified; typically activations
  Model weights                   7.48 GiB    7.1%      +0.00 GiB   parameters and buffers of the loaded model(s)
  Communication buffers           2.01 GiB    1.9%      +0.00 GiB   multi-GPU communication buffers
                                of which (GiB): NCCL communicators 0.94, NCCLWindowAllocator 0.56, custom allreduce workspace 0.50
  CUDA and library overhead       1.28 GiB    1.2%      +0.06 GiB   CUDA context, cuBLAS, NCCL internals and other driver memory
                                of which (GiB): before the first startup step 0.60, Additional executor resources (temporary for KV cache size estimation) 0.38, Model 0.10, Sampler 0.07, Model resources created during usage 0.06, +1 more
  Runtime buffers                 0.23 GiB    0.2%      +0.00 GiB   buffers owned by named runtime components
                                of which (GiB): Additional executor resources 0.10, Model 0.06, Additional executor resources (temporary for KV cache size estimation) 0.03, Model resources created during usage 0.03
  CUDA graphs                     0.05 GiB    0.1%      +0.00 GiB   held by captured CUDA graphs for replay; not reusable
                                of which (GiB): in use by graphs 0.03, free in graph pools 0.02
  PyTorch cache (free)            0.01 GiB    0.0%      -0.72 GiB   held by the PyTorch allocator but not in use; reusable

  Startup plan: peak outside the KV cache 11.68 GiB + KV cache 83.03 GiB = 94.71 GiB; device now 106.25 GiB, excluding 0.01 GiB of free PyTorch cache (+11.55).
    of which the profiled forward pass allocated 0.67 GiB in PyTorch.
  Usage exceeds the startup plan by 11.55 GiB: the workload needs more than profiling measured, or another process grew.

  Conclusion: since warmup, PyTorch, unattributed grew the most (+12.00 GiB).
  Suggestion: PyTorch memory with no known owner grew 12.00 GiB after warmup, 11.33 GiB more than the profiled forward pass allocated (0.67 GiB), so something may be holding on to memory (a cache, a callback, a leak); rerun with TLLM_MEM_MONITOR_STACKS=1 to name the code that allocated it; to reduce activation memory, reduce max_num_tokens.

The 12.00 GiB is the leak (3 x 4 GiB), and the suggestion separates it from what the profiled forward pass needs. In the default mode the same report has every row and the same total, but no "since warmup" column, no startup plan and no per-row breakdown.

  • Groups add up exactly. Every counted entry belongs to one group named for its purpose. The groups sum to the process total, and a test fails if any category has no group.

  • Growth is measured from warmup. Changes are shown against the report taken when startup completes, since every static allocation exists by then. A failed startup does not become the baseline.

  • Advice comes from the executor. Suggestions are the executor's own text, reused from _ExecutorMemoryMonitor, and appear only for an OOM, including a startup that ran out of memory.

  • It compares against the startup plan. KV cache sizing records its plan (the peak outside the KV cache), and current usage is compared with it. Free PyTorch cache is excluded from that comparison: sizing measured after emptying the cache, and the workload can reuse that space.

  • Startup steps are measured per process. Each executor creation step is measured through NVML. The executor's own figures are device-wide; on a shared GPU, two runs of the same model reported the weight-loading step as 1.08 GiB and 0.06 GiB.

  • Optional allocation stacks. With TLLM_MEM_MONITOR_STACKS=1 (off by default) the "PyTorch, unattributed" row is broken down by the code that allocated each block: the innermost TensorRT-LLM frame, or the third-party library. It records allocator state rather than an event history. Measured on torch 2.14, allocation got 26% slower and a snapshot took 6 ms; keeping the full history instead costs 47% and 2.3 s.

  • Sleep mode is accounted for. Once release_with_tag has run, torch still counts the released pools while the GPU no longer backs them. The monitor leaves released pools, and a released KV cache, out of the PyTorch footprint and out of every row, so the rows show only resident memory and still add up; one line lists what is released and returns on resume. A released tag counts only for the monitored executor's own sleep-mode memory, so another executor's sleep does not hide its resident pools. On a real release of the weights and model pools the process dropped from 19.63 GiB to 18.65 GiB. The overhead row stayed at 1.00 GiB; without the fix it would have fallen to about 0.

  • Reports while serving. With TLLM_MEM_MONITOR=1 and TLLM_MEM_MONITOR_INTERVAL=<seconds>, a daemon thread reports at that interval once startup is over, so memory that grows slowly is visible before it runs out.

  • Reports on request. LLM.get_memory_report() (prototype API, also on AsyncLLM) returns one entry per rank, rank 0 first, in every mode: a report, or a line saying why that rank has none.

    • With the Ray executor every rank answers directly.
    • With the MPI executor in full mode, rank 0 gathers the other ranks' reports over a communicator duplicated at startup and freed at shutdown (every rank reads the same mode from the arguments; MPI must provide MPI_THREAD_MULTIPLE; the KV-sizing pass gets none). A listener thread on each rank polls for requests and replies without blocking; a rank that does not answer within 30 s is reported as such. In the default mode the other ranks are listed as not gathered.
    • A trtllm-serve with memory_monitor_config.mode: full (or TLLM_MEM_MONITOR=1), a runtime-control key (TRTLLM_RUNTIME_CONTROL_API_KEY or the server's runtime_control_api_key) and a single serving frontend also serves GET /memory_report, with the runtime-control request signing like /memory_status; sleep mode and the runtime-control endpoints are not needed. Otherwise the route is not served and the server logs why. It is signed because a report exposes PIDs, other processes' usage and, in stack mode, source paths. Concurrent requests share one gather, and on-demand reports are not written to files. Weights are always claimed (a weak reference only), so a report names them in every mode.
  • OOMs from any thread, reported once they are failures. torch's OOM observer only records the failed allocation (size, free memory, what PyTorch held); it builds no report, because callers such as warmup, KV sizing and optional workspaces catch OOMs and recover. The report is built when the error escapes. In every mode but off, the executor's own error exits report it: _handle_errors (after the error responses are sent), a crashed _event_loop_wrapper (after its crash watchdog is armed; the watchdog now starts in the except block), and virtual_memory.materialize_with_tag, which reports every wakeup failure whoever called it. In full mode an ERROR log record (built on its own short-lived thread; background threads such as the KV transfer workers log their failures) and threading.excepthook report it too, and torch's OOM observer, installed only in full mode, puts the failed allocation into the headline. Errors are classified by exception type where one is at hand (tensorrt_llm._startup_errors.out_of_memory_kind, shared with startup error classification and the executor's OOM explanation), by text otherwise: host-memory failures (cudaHostRegister, cuMemHostAlloc, pinned memory, HostOOMError) never count, and failures that do not say which memory ran out (CUDA's generic out of memory, CuOOMError, NCCL's unhandled cuda error) count only on a nearly full device. Reports are built on the executor's device, from any thread.

  • Allocations by worker threads during startup. Tags are per thread (measured), so each tag's time window is also kept. An untagged allocation event inside a window takes the innermost window's label; explicit tags and claimed modules still take precedence.

  • Suggestions use growth since warmup. Warmup already runs the largest batch, so PyTorch memory with no known owner that grows past it means something is holding memory. The suggestion says so: it names the allocating code if stacks were recorded, and otherwise suggests TLLM_MEM_MONITOR_STACKS=1. Growth of 1 GiB or more is explained in periodic reports too.

  • Reports can be kept as files. With report_dir / TLLM_MEM_MONITOR_DIR set, every report except on-demand ones is also written to its own file, in the default mode as well as full mode, named by rank, PID, UTC time and tag; the directory and files are private to their owner, and each process keeps the newest 100 periodic files it wrote. Logs from several ranks interleave and are often cut short when a job dies.

Below the summary is a section broken down by allocator, plus conservation checks that compare independent sources, so a failure means the sources disagree.

How memory is attributed

Memory Source
KV cache v1 get_kv_cache_stats().allocated_bytes; on Qwen2-0.5B this matched the engine's estimate, 17.54 GiB.
KV cache v2 (target, draft, cross) StorageStatistics. The managers register themselves where they are built and are held weakly, so the monitor never keeps a pool alive. Only device-backed pools count; with sparse KV cache offload, pools of pinned host memory mapped into the device are listed separately as host-backed and not counted.
Weights The claimed model's current parameters, matched to allocator blocks. Shared storage such as tied embeddings is counted once. Replaced weights are followed.
Executor creation steps Each ExecutorMemoryType step tags what it allocates. On Qwen2-0.5B, torch memory nobody could name fell from 0.17 GiB to 0.03 GiB.
CUDA graphs Blocks in a graph's private pool, live or free. Graph pools and MemPools share an id counter on torch 2.14, so the MemPool creation sites register their pools. A MemPool nobody registered, such as the one behind GMS weight sharing, is found as a live MemPool object and is not mistaken for a graph pool.
Sleep-mode pools Named by their tag, including the pool that resumes a parent scope after a nested one.
MNNVL, DeepEP, DWDP pages, DWDP local weight shards The components' own registries; DWDP shards moved out of PyTorch into fabric handles count as weights
SymmetricMemoryAllReduce Found by walking the claimed model. Measured on torch 2.14: these buffers do not show up in memory_reserved().
MNNVL/NVLS allreduce multicast MNNVLAllReduce's class-level workspaces, both active and retired
NCCL window buffers NCCLWindowAllocator::getDeviceUsage, a new accessor read through trtllm::get_nccl_window_usage
Native allocations in C++ A small ledger (common/nativeMemory.h) that allocation sites record into by label, read through trtllm::get_native_memory_usage. Its labels are:
• NCCL communicators. NCCL reports no size, so the drop in free memory across ncclCommInitRank is recorded, and a reading that comes out negative (memory freed elsewhere meanwhile) makes the size unknown rather than wrong; with NCCL_RUNTIME_CONNECT=0, that is where NCCL allocates its connection buffers.
• The fused GEMM+allreduce NVLS workspace, exact size.
• IPC buffers (custom allreduce, low-precision allreduce, TP peer exchange), exact size.
• The MoE GEMM profiler workspace, exact size, held while autotuning.
• Async Ulysses slots, exact size.
• Userbuffers (the torch.compile allreduce buffers), exact registered size, kept until exit.
• EPLB host-accessible memory, the requested size, labelled by backing; only the GDRCopy path is summed, and the NUMA path keeps its pages until exit and records no release.
Libraries allocating outside PyTorch The overhead row is split by when each part appeared: before the first startup step (CUDA context), each executor step, and named regions around library init (LoRA device cache, KV transfer buffers, communicators created by torch.distributed). Regions nest without double counting, and those of the temporary KV-sizing executor are dropped when it is torn down. The parts sum to the row exactly.

At TP=2 on Llama-3.1-8B, "Communication buffers" reads NCCL communicators 0.94 GiB, NCCL window 0.56 GiB and the custom allreduce workspace 0.50 GiB per rank, all named in every mode.

Reporting is armed once the monitored executor has been created, and suspended while it is being created, since KV sizing, warmup and graph capture deliberately run allocations into OOM and catch the error. The monitor follows one executor per process: a later executor in the same process is not monitored (no settings change, arming, startup record or providers, and its get_memory_report() says why) and does not suspend reporting. Shutdown releases everything the monitor holds for the executor through one call, providers.end_executor(token), also when the executor loop ends without shutdown() or an earlier shutdown step raises.

PagePool also gains held_pages(), plus a process-wide count of bytes whose release failed (unreleased_page_bytes()).

Not in this PR

  • NIXL bounce arena (C++ transfer agent): it lives in a wrapper library that does not link libtensorrt_llm, so it cannot use the ledger. It is created with the transceiver and is measured inside the "KV cache transfer buffers" region.
  • CUTLASS MoE scratch: this is torch memory allocated lazily inside a C++ op, so it shows as "PyTorch, unattributed", or by allocating code with TLLM_MEM_MONITOR_STACKS=1.
  • Smaller gaps: NCCL buffers that torch.distributed connects lazily after its first collective, and JIT-loaded module code, land in the overhead row, split by the startup step in which they appeared. Communicator memory is a device-wide measurement, so a concurrent allocation by another process on the same GPU would be counted too.
  • get_rank_model_storage (pre-existing): it over-counts tied embeddings, reporting 1.17 GiB against 0.92 GiB actually held on Qwen2-0.5B.

Follow-ups (not in this PR)

These came up in review and are deliberately left to follow-up PRs so that this one does not grow further:

  • Typed per-rank report API. LLM.get_memory_report() and GET /memory_report return rendered text, one string per rank. A typed per-rank result would mirror get_memory_status: rank, pid, device, status, totals, groups and checks, plus the text. MonitorCapture already holds that structure. (review: discussion_r4551338999)
  • Provider protocol and registration where memory is allocated.
    • Declare the optional provider methods (describes(), collect_with_pool_rows()) on MonitorProvider with defaults.
    • Tell KV cache v1 and v2 apart without hasattr.
    • Let components register their providers where they allocate, instead of default_providers() reading other components' internals and the creator registering the KV pools.
    • This way a new cache manager or buffer cannot silently fall into the overhead row. (discussion_r4551339020)
  • One per-step memory record. _ExecutorMemoryMonitor measures each executor creation step device-wide, and the monitor measures it per process. A single per-step record would let the startup-OOM explanation and the monitor share one measurement: per-process NVML readings in every mode, falling back to device-wide. (discussion_r4551339031)
  • One KV sizing plan object. KV sizing would produce the plan, including every reserve taken from the KV budget, and both the capacity code and the monitor would read it, instead of the call site re-deriving kv_budget_reserves. (discussion_r4551339042)
  • On-demand VisualGen report. VisualGen workers report out-of-memory errors, startup and periodic reports. VisualGen.get_memory_report() and a trtllm-serve --visual_gen_args route would be a public VisualGen API change, so they need the VisualGen team sync first. (discussion_r4551339010)
  • OOM-time allocator snapshot. In full mode the OOM observer records only the failed request and the allocator counters; the snapshot behind the breakdown is taken later, when the error reaches the executor's handler (now on a background thread). Temporaries freed while the exception unwinds, such as MoE workspaces, can then be missing from the breakdown although they are in the counters. Retain an allocator snapshot inside the observer, before unwinding, and have the report prefer it; offered as an opt-in diagnostic mode because of its cost. It only covers OOMs raised by the PyTorch caching allocator.
  • Opt-in stack recording during startup. Startup records allocation history without stacks, only to tag allocations by startup step, and TLLM_MEM_MONITOR_STACKS=1 starts after a successful start. A warmup OOM (seen on DeepSeek-V4-Pro) can therefore be pinned to the step but not split into MoE scratch, attention buffers and activations. Add an opt-in mode that records stacks from model loading on, with its overhead documented (Python stacks alone slowed allocations by about 26%), and reconcile it with the startup-step tags.
  • EPLB NUMA path on hardware.
    • The NUMA branch of the EPLB host-accessible allocator, which deliberately records no release, has not run locally.
    • Its test is listed in the GB300 pre-merge stage (l0_gb300.yml, GB300-4_GPUs-PyTorch-1), so CI runs it first.

Performance

  • Default (oom). No added code runs per iteration, per forward or per allocation. Nothing is installed process-wide; the executor's error exits call the monitor when an error reaches them. Startup hooks cost about 1.6 µs each and take no allocator snapshot. The C++ ledger records only when native buffers are allocated or released. The process's memory pools are classified once when the executor has finished starting (one walk of the Python heap, under a second for 3M objects), so a report built during an OOM does not pay for it.

  • Off (0). As the default, without reporting.

  • Full (1). Arming installs PyTorch's OOM observer, a filter that acts only on ERROR log records, and a threading.excepthook; all three do nothing until something fails. Nothing runs per iteration either. The periodic reporter only wakes on its timer. Allocation history is recorded only during startup and is stopped when startup ends. Reports are cheap: one allocator snapshot each, and each pool is classified once. With 3M Python objects and a CUDA graph alive, the first report took 0.81 s, because it walks the heap once to classify pools, and later reports about 1 ms; before this change they took 0.14-0.18 s each.

  • Throughput. Measured on Qwen2-0.5B, B200, 128 x 128 tokens, which is host-bound and so the most sensitive case. Ten interleaved runs each, pinned to idle cores: off averaged 24,100 tok/s and full 23,876 tok/s. The pairwise difference is -0.8% (range -5.5% to +3.5%), which is not distinguishable from noise. Against the base commit, which lacks this feature, the off runs measured 24,890-25,466 tok/s versus 24,939-25,192 tok/s. The default mode does a subset of the full mode's work and was not measured separately.

  • Stack mode. TLLM_MEM_MONITOR_STACKS=1 is the one mode with a steady-state cost: allocations are 26% slower in a microbenchmark, and the same run measured 24,675 tok/s. It is off by default and is documented as such.

  • Measured at scale.

    • Token latency around a report (H100 SXM and H100 NVL; Llama-3.1-8B-Instruct, TP=2, CUDA graphs, 32 concurrent streaming requests, OSL 512; get_memory_report() at 30/50/70 s; two rounds per mode). Across 24 calls the largest inter-token gap during a call was 13.7 ms, never above the largest gap the same run had without a call (12-15 ms; p99 5.7-7.4 ms). The first call is not slower than later ones, since pools are classified when the executor has finished starting. A report took 10-32 ms in the default mode and 113-264 ms in full mode, where rank 0 gathers the other rank's report on the caller's thread.

    • Startup (H200 x8; each model four times on one node, off/full/full/off; compared on the warm-cache pair):

      Model Startup off / full Peak RSS per rank off / full
      Llama-3.1-70B-Instruct, TP8 90.9 s / 92.3 s (+1.5 %) 102.8 / 103.2 GiB (+0.35-0.4)
      DeepSeek-V3 (FP8), TP8 EP8 194.1 s / 194.0 s (0 %) 31.5-31.95 / 32.07 GiB (+0.1-0.6)

      get_memory_report() at 70B TP8 took 0.17-0.45 s in full mode, gathering 8 ranks.

Risk

By default the monitor installs nothing process-wide and only reports out-of-memory errors that reach the executor's error handling; mode off turns even that off. Outside full mode, model loading takes no allocator snapshot and the tags touch no allocator state. Every provider, check and report is wrapped so that a failure is logged and skipped. The OOM handler allocates no device memory and swallows its own errors, so it can never replace the OutOfMemoryError.

Outside the monitor package the change is a set of call sites:

  • one line in model_loader.py
  • the startup window, OOM arming and step hooks in py_executor_creator.py
  • KV registration and plan recording in _util.py
  • the error hooks in py_executor.py; also there, the sleep/wakeup ACK wait now uses the shared bounded receive in tensorrt_llm/_torch/distributed/mpi_polling.py (_sleep_wakeup_ack_ready removed); the monitor teardown runs from shutdown() (in a finally) and at the end of _event_loop_wrapper; resource managers leave the monitor right before they are shut down; the crash watchdog in _event_loop_wrapper starts in the except block, before the error is reported and logged. No collective runs for the monitor: every rank reads the mode from the arguments.
  • trtllm-serve: the /memory_report route, and /memory_status now shares its dispatch helper (_call_runtime_memory_method); the server reads memory_monitor_config from the arguments and does not configure the monitor; the route snapshot gains llm_memory_report, which needs API review committee sign-off
  • LLM.get_memory_report / AsyncLLM.get_memory_report (prototype; tests/unittest/api_stability/references/llm.yaml updated), BaseWorker.get_memory_report (an unmonitored engine answers with a "not monitored" line per rank), and the MPI multi-rank RPC allow-list in executor/proxy.py
  • pool registration and release tracking in virtual_memory.py; pool registration in locality_domain_utils.py
  • named regions in distributed/communicator.py, distributed/ops.py, _util.py (KV transfer) and resource_manager.py; exact IPC buffer sizes in tensorrt_llm/_ipc_utils.py
  • tensorrt_llm/_startup_errors.py: out_of_memory_kind, the shared OOM classifier
  • pyexecutor/hang_detector.py: rank_crash_kill_pending(), so a failed startup skips its report while a crash kill is pending
  • VisualGen: _torch/visual_gen/memory_monitor.py and its hooks in _torch/visual_gen/executor.py (internal only; the public VisualGen API is unchanged)
  • memory_monitor_config in llm_args.py (prototype; golden manifest and serve schemas regenerated), passed to the monitor by the executor creator and trtllm-serve
  • C++: the ledger (common/nativeMemory.h/.cpp, its own static component so component-local gtests link) and its record calls, including one per registered userbuffer (kernels/userbuffers). These are a map update under a mutex on allocation and release only, never on the hot path. The label namespace is nativeMemoryLabels, and adjustNativeAllocation lowers a record's bytes without dropping its count. Also NCCLWindowAllocator::getDeviceUsage, which reads per-device totals under their own lock and never waits for an allocation or window registration, and three host-side torch ops (in thop/nativeMemoryOp.cpp).
  • the PagePool accessors

Test Coverage

The monitor tests run in CI. tests/unittest/_torch/memory has 179 test functions, 228 test cases with a GPU; 192 of them are cpu_only, so the CPU stage runs them, and unittest/_torch/memory is listed in l0_h100, whose not cpu_only selection runs the rest. 50 more cases cover the executor hooks (tests/unittest/_torch/pyexecutor/test_executor_memory_monitor_hooks.py), VisualGen (tests/unittest/_torch/visual_gen/test_executor_memory_monitor.py), the serve route (tests/unittest/serve/test_memory_report_route.py), the MPI report (full_config and default) and mpi_polling. The page-pool test runs in l0_b200 next to the other DWDP GPU tests, the two-rank MPI report test is collected by the existing unittest/_torch/multi_gpu entry, and the C++ ledger has a gtest (NativeMemoryTest). The Ulysses slot-ledger test is listed in the multi-GPU l0_dgx_b200 block, the 2-GPU userbuffers ledger test runs with the existing test_user_buffers.py entry, and the EPLB native test is also listed in the GB300 pre-merge stage, where the allocator takes the NUMA path. Before this, none of the monitor's test files ran in CI: the CPU stage skips unmarked files, and no GPU stage listed the directory.

User documentation: docs/source/features/memory-monitor.md.

pytest tests/unittest/_torch/memory/
pytest tests/unittest/_torch/modules/dwdp/test_page_pool_statistics.py

228 monitor test cases in tests/unittest/_torch/memory plus the 50 above (tests that check one behaviour are merged, each case named in its assertion message), plus a 2-GPU Ulysses test and the trtllm-serve route tests. Real CUDA graph capture, the real sleep-mode pool with a real release_with_tag, real stack recording, a real MoE autotuning run and the real EPLB allocator are used where the behaviour depends on torch. Each rule that a measurement established has a test that fails when the rule is violated; this was verified by reverting each rule in turn. Some examples:

Reverted to Test that fails
Counting graph-pool free space as reusable test_graph_memory_is_cuda_graphs_not_reusable_cache
Arming before startup, or arming a second executor test_only_the_first_executor_in_a_process_is_monitored
Ignoring OOMs reported by the executor, or reporting before the error responses test_error_handler_reports_after_the_responses
Claiming weights by address at load time test_freed_weights_do_not_label_their_successor
Holding the KV manager strongly test_manager_is_held_weakly
A category with no summary group test_groups_add_up_and_every_category_has_a_group
Restating instead of reusing executor advice test_executor_memory_monitor_feeds_the_report
Check C reading derived properties test_check_c_can_fail_and_does_not_read_the_derived_properties
Treating an unregistered MemPool as a graph pool test_pools_are_classified_by_kind
Counting released sleep-mode memory as held test_sleep_mode_pools_are_named_and_released_memory_leaves_the_footprint
Registering a resumed parent scope without its tag test_sleep_mode_pools_are_named_and_released_memory_leaves_the_footprint
Removing the C++ record calls (rebuilt) test_moe_gemm_profiler_workspace_is_recorded_while_profiling, test_eplb_host_accessible_memory_is_recorded
Counting a nested region twice test_regions_count_each_byte_once
Listing a region entered twice as two parts test_regions_count_each_byte_once
Taking two snapshots per report, or walking the heap on every report test_report_cost
Reading live from the allocator counters instead of the snapshot test_groups_add_up_when_allocator_counters_lag_the_snapshot
Keeping one executor's startup state for the next test_startup_state_does_not_leak
Arming nothing (or everything) when TLLM_MEM_MONITOR is unset TestDefaultReportsOnlyOutOfMemory
Removing the LoRA, KV transfer or listener call site test_lora_device_cache_is_named, test_kv_transfer_buffers_are_named, test_sleep_listener_reports_a_failed_wakeup_once
Installing global hooks in the default mode test_the_default_mode_installs_no_process_wide_hook
Counting host-backed KV pools as device memory test_host_backed_pools_are_not_counted_as_device_memory
Another executor's sleep releasing the monitored KV cache test_another_executors_sleep_does_not_release_the_monitored_kv_cache
Releasing the monitor only on the late shutdown path test_a_loop_that_ends_without_shutdown_leaves_the_monitor
Keeping the duplicated communicator test_stopping_frees_each_ranks_communicator
Reading NCCL window usage under the allocator lock UsageIsReadWithoutWaitingForAnAllocationInProgress
A frozen startup label on a block with frames test_a_block_with_frames_never_takes_a_frozen_startup_label
Classifying host-memory failures as GPU OOMs test_other_errors_and_host_memory_are_not_reported, test_an_exception_is_classified_by_its_type

Regression on the touched files, at the head of this PR:

  • _torch/memory: 228 passed with a GPU, and 192 cpu_only passed without one.
  • Executor hooks, test_py_executor.py, test_py_executor_creator_*, KV estimation, test_connector_adp, mpi_polling, startup errors, virtual memory, DWDP page pool and VisualGen: 855 passed.
  • test_dynamic_eplb_contracts.py: 53 passed.
  • tests/unittest/serve (cpu_only), the multi-frontend gate and the serve config schemas: 429 passed. The one failure, test_no_object_new, flags sites in other serve tests that this PR does not touch; it is waived (nvbugs/7081838).
  • test_llm_args.py and api_stability: 709 passed.
  • C++: ctest -L common 39/39 and NativeMemoryTest 6/6.
  • 2 GPUs: the ncclUtilsTest window tests 45/45 per rank; the MPI report, userbuffers ledger and Ulysses tests 8 passed.

End-to-end runs on Qwen2-0.5B-Instruct (B200) gave the following:

  • The KV pool matches the engine's own estimate: 17.44 GiB.
  • Checks report [checks] 3 passed, 0 FAILED.
  • No startup OOM was falsely reported.
  • Startup with TLLM_MEM_MONITOR=1 was not measurably slower than with the monitor off.
  • A real OOM produced the report shown above, and the OutOfMemoryError still propagated.
  • A sleep/wakeup cycle with TLLM_MEM_MONITOR=1 produced the same outputs before and after.
  • TP=2 with TLLM_MEM_MONITOR_INTERVAL=5 and TLLM_MEM_MONITOR_DIR wrote each rank's startup and periodic reports; rank 1's periodic reports name device 1. llm.get_memory_report() returned rank 0's report.
  • Without TLLM_MEM_MONITOR=1, llm.get_memory_report() returned a report that names the model weights (0.92 GiB).
  • TP=2 (two GPUs) reports NCCL communicators and NCCL window buffers under "Communication buffers"; checks pass 3/3.

Modes, on B200:

  • A forced OOM in one process with TLLM_MEM_MONITOR unset, oom, 0 and 1 gave one OOM report, one, none, and one plus periodic reports.
  • Llama-3.1-8B-Instruct at TP=2 with the leak shown above: with the variable unset, each rank wrote one OOM report and nothing else; with 1, each rank wrote a startup report and an OOM report. Serving continued in both.

Real runs after rebasing on main, with each report checked for exact sums and passing checks:

  • B200: the scenario matrix (monitor off, TLLM_MEM_MONITOR=1, KV cache V1, a real serving OOM, stack mode, sleep and wake, a startup that fails beside a colocated process, TP=2, a rank filter). Also Nemotron-H-8B (hybrid Mamba), and trtllm-serve GET /memory_report (HTTP 200).
  • H100, H200, A100, L40S: dense, FP8, MoE (TP2+EP2), MLA (DeepSeek-V3-Lite TP2) and Phi-4-mini, using a build for sm80, sm89, sm90 and sm100.
  • Weights row: matches the checkpoint size divided by the parallelism on every GPU.

The fused GEMM+allreduce NVLS record call is compiled but not exercised. Its NVFP4 model needs an FP8-KV attention kernel that FAST_BUILD omits, and the JIT fallback for it cannot find cuda.h in this environment.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

🤖 Generated with Claude Code

@zhaoyangwang-nvidia zhaoyangwang-nvidia changed the title [None][feat] Add a GPU memory ledger with basis-tagged attribution [None][feat] Add a global memory monitor Oct 27, 2026
@zhaoyangwang-nvidia zhaoyangwang-nvidia changed the title [None][feat] Add a global memory monitor [TRTLLM-17451][feat] Add a global memory monitor Oct 28, 2026
@zhaoyangwang-nvidia
zhaoyangwang-nvidia force-pushed the feat/memory-ledger branch 5 times, most recently from 6f8c40b to c338df6 Compare November 2, 2026 06:10
@zhaoyangwang-nvidia zhaoyangwang-nvidia added the api-compatible Accepted LLM API contract change that is backwards-compatible label Nov 2, 2026
@zhaoyangwang-nvidia
zhaoyangwang-nvidia marked this pull request as ready for review November 9, 2026 13:04
@zhaoyangwang-nvidia
zhaoyangwang-nvidia requested review from a team as code owners November 9, 2026 13:04
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86233 [ run ] triggered by Bot. Commit: 23ebd06 Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86251 [ run ] triggered by Bot. Commit: 8647a87 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86233 [ run ] completed with state ABORTED. Commit: 23ebd06

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86251 [ run ] completed with state SUCCESS. Commit: 8647a87
/LLM/main/L0_MergeRequest_PR pipeline #71456 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86305 [ run ] triggered by Bot. Commit: 8c5cc10 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86305 [ run ] completed with state SUCCESS. Commit: 8c5cc10
/LLM/main/L0_MergeRequest_PR pipeline #71504 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86383 [ run ] triggered by Bot. Commit: 8c5cc10 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86383 [ run ] completed with state SUCCESS. Commit: 8c5cc10
/LLM/main/L0_MergeRequest_PR pipeline #71580 completed with status: 'SUCCESS'

CI Report

Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86489 [ run ] triggered by Bot. Commit: 1ad0d4e Link to invocation

…buffers

Add a C++ native memory ledger (common/nativeMemory) that records exact
sizes for allocations PyTorch cannot see, and expose it through the
trtllm::get_native_memory_usage / get_native_memory_labels ops:
NCCL communicators and window buffers (per-device usage kept outside the
allocation lock), NVLS multicast and fused GEMM+allreduce buffers, the MoE
load balancer, Ulysses all-to-all slots and userbuffers.

Track CUDA IPC buffers in IpcMemory, including buffers a caller allocated
itself (PeerBuffers), and name the buffers the distributed helpers create.
Move the MPI polling helper used by sleep/wakeup acknowledgements and the
rank report gather to _torch/distributed.

Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add tensorrt_llm/_torch/memory/monitor, which explains where a process's
GPU memory went: NVML process total = PyTorch memory + known non-PyTorch
memory + residual overhead, broken down by owner (weights, KV cache pools,
CUDA graph pools, activations, communication buffers, native allocations).

- Three modes: off, oom (default: a report when the executor's own error
  handling sees a GPU out-of-memory error, nothing installed process-wide)
  and full (startup record per executor step, warmup baseline and plan,
  periodic and on-demand reports, optional allocation stacks).
- Providers for the KV cache (device-backed pools only), MNNVL, DeepEP,
  symmetric memory, multicast allreduce, IPC, NCCL and the native ledger,
  with conservation checks against the allocator and NVML.
- One out-of-memory classifier, shared with startup error classification,
  that tells device, host and disk failures apart by exception type.
- The monitor follows the first live executor in a process.

Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
…ualGen integration

- PyExecutor and ScheduledEncoderExecutor register with the monitor at
  creation, record startup steps, arm OOM reporting once started and
  release the monitor on every exit path; KV cache managers are registered
  where they are built and dropped before they are freed.
- TorchLlmArgs.memory_monitor_config (prototype) carries the settings, with
  the TLLM_MEM_MONITOR* environment variables folded in once; the golden
  manifest and serve config schemas are regenerated.
- LLM.get_memory_report() gathers a report from every rank; trtllm-serve
  serves GET /memory_report in full mode with a runtime-control key and a
  single frontend, pinned in the route snapshot.
- VisualGen diffusion workers report OOMs from their error exit and record
  the pipeline load in full mode.

Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add the GPU and multi-GPU memory monitor tests to the B200, H100 and GB300
pre-merge lists and the multi-GPU trigger list.

Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add the memory monitor feature page and link it from the serve and
sleep/wakeup pages.

Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86500 [ run ] triggered by Bot. Commit: 346107c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86489 [ run ] completed with state ABORTED. Commit: 1ad0d4e

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86500 [ run ] completed with state SUCCESS. Commit: 346107c
/LLM/main/L0_MergeRequest_PR pipeline #71688 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@zhaoyangwang-nvidia

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86521 [ run ] triggered by Bot. Commit: 346107c Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #86521 [ run ] completed with state SUCCESS. Commit: 346107c
/LLM/main/L0_MergeRequest_PR pipeline #71707 completed with status: 'SUCCESS'

CI Report

Link to invocation

@zhaoyangwang-nvidia
zhaoyangwang-nvidia merged commit 2bbe880 into NVIDIA:main Nov 21, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-compatible Accepted LLM API contract change that is backwards-compatible ci: full pre-merge approved

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants