Repository navigation
[TRTLLM-17451][feat] Add a global memory monitor - #21113
Conversation
9b5bfdd to
ae0aa66
Compare
6f8c40b to
c338df6
Compare
fa5b79b to
9c14e7f
Compare
|
PR_Github #86233 [ run ] triggered by Bot. Commit: |
23ebd06 to
8647a87
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #86251 [ run ] triggered by Bot. Commit: |
|
PR_Github #86233 [ run ] completed with state |
|
PR_Github #86251 [ run ] completed with state
|
8647a87 to
8c5cc10
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #86305 [ run ] triggered by Bot. Commit: |
|
PR_Github #86305 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #86383 [ run ] triggered by Bot. Commit: |
|
PR_Github #86383 [ run ] completed with state |
8c5cc10 to
1ad0d4e
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #86489 [ run ] triggered by Bot. Commit: |
…buffers Add a C++ native memory ledger (common/nativeMemory) that records exact sizes for allocations PyTorch cannot see, and expose it through the trtllm::get_native_memory_usage / get_native_memory_labels ops: NCCL communicators and window buffers (per-device usage kept outside the allocation lock), NVLS multicast and fused GEMM+allreduce buffers, the MoE load balancer, Ulysses all-to-all slots and userbuffers. Track CUDA IPC buffers in IpcMemory, including buffers a caller allocated itself (PeerBuffers), and name the buffers the distributed helpers create. Move the MPI polling helper used by sleep/wakeup acknowledgements and the rank report gather to _torch/distributed. Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add tensorrt_llm/_torch/memory/monitor, which explains where a process's GPU memory went: NVML process total = PyTorch memory + known non-PyTorch memory + residual overhead, broken down by owner (weights, KV cache pools, CUDA graph pools, activations, communication buffers, native allocations). - Three modes: off, oom (default: a report when the executor's own error handling sees a GPU out-of-memory error, nothing installed process-wide) and full (startup record per executor step, warmup baseline and plan, periodic and on-demand reports, optional allocation stacks). - Providers for the KV cache (device-backed pools only), MNNVL, DeepEP, symmetric memory, multicast allreduce, IPC, NCCL and the native ledger, with conservation checks against the allocator and NVML. - One out-of-memory classifier, shared with startup error classification, that tells device, host and disk failures apart by exception type. - The monitor follows the first live executor in a process. Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
…ualGen integration - PyExecutor and ScheduledEncoderExecutor register with the monitor at creation, record startup steps, arm OOM reporting once started and release the monitor on every exit path; KV cache managers are registered where they are built and dropped before they are freed. - TorchLlmArgs.memory_monitor_config (prototype) carries the settings, with the TLLM_MEM_MONITOR* environment variables folded in once; the golden manifest and serve config schemas are regenerated. - LLM.get_memory_report() gathers a report from every rank; trtllm-serve serves GET /memory_report in full mode with a runtime-control key and a single frontend, pinned in the route snapshot. - VisualGen diffusion workers report OOMs from their error exit and record the pipeline load in full mode. Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add the GPU and multi-GPU memory monitor tests to the B200, H100 and GB300 pre-merge lists and the multi-GPU trigger list. Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
Add the memory monitor feature page and link it from the serve and sleep/wakeup pages. Signed-off-by: ZhaoyangWang <265160349+zhaoyangwang-nvidia@users.noreply.github.com>
1ad0d4e to
346107c
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #86500 [ run ] triggered by Bot. Commit: |
|
PR_Github #86489 [ run ] completed with state |
|
PR_Github #86500 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #86521 [ run ] triggered by Bot. Commit: |
|
PR_Github #86521 [ run ] completed with state |
JIRA: TRTLLM-17451
Description
Why
When a rank runs out of memory today, all it reports is the out-of-memory error. It does not say which memory filled the device, what changed since startup, or what to adjust. The pieces that could answer this are spread over several allocators, and they do not add up naively. Measured on B200:
MemoryCountersoverlapstorch_reservedrather than adding to it. A 512 MiB allocation raises both by 512 MiB, so summing the two doubles the figure.PagePool.num_pages()reports configured capacity, not occupancy. It still reported 128 MiB after the driver had taken all of it back.MAPPED. A rollup that counts onlyMAPPEDreports zero during an OOM.Why one PR
The report is only useful if its rows add up to what the process holds: a row that cannot be trusted to be complete sends the reader to the wrong component, which is worse than no report. That makes the data sources one unit -- the PyTorch pools, the KV cache managers, the C++ ledger, the communication buffers and the startup steps each close a gap the others leave, and leaving one out turns its memory into unexplained overhead. The on-demand API, the MPI gathering and the serve route are thin readers of the same report. The C++ side is a small ledger plus record calls at the sites that allocate outside PyTorch. The docs page describes the whole.
What it does
The monitor is configured with the prototype
memory_monitor_configin the LLM arguments (mode:oomby default,fulloroff;ranks,record_stacks,periodic_report_interval_s,oom_report_interval_s,report_dir). Each field has aTLLM_MEM_MONITOR*environment variable that overrides it, so a run can be diagnosed without changing its configuration; the variables are folded into the arguments once, where they are built, and reach every rank insidememory_monitor_config. The defaultoommode reports an out-of-memory error that reaches the executor's own error handling (a failed request, a crashed executor loop, a failed wakeup) and installs nothing process-wide;fullalso records startup, installs process-wide hooks so errors on background threads and logged errors are reported too, reports once startup has completed (warmup included) and compares later reports with that point;offproduces no reports on its own. The monitor follows one executor per process, the first one alive; another executor in the same process runs without it, with one warning. Out-of-memory reports are logged at ERROR, so they are visible at the default log level. The summary comes first and is written so a reader does not need to know which allocator holds what.memory_monitor_config.mode/TLLM_MEM_MONITORoom/ unsetLLM.get_memory_report()works; under MPI only rank 0 answers.full/1threading.excepthook, torch OOM observer). Optional periodic reports and stack recording; every rank gathered under MPI;GET /memory_reportwith a runtime-control key and a single frontend.off/0LLM.get_memory_report()still answers on request.Every allocator and every component that owns GPU memory feeds one monitor, which places each byte in exactly one group named for its purpose:
flowchart LR subgraph SRC["Allocators and components"] direction TB S1["PyTorch caching allocator<br/>one snapshot per report, split by pool:<br/>default, CUDA graph, MemPool"] S2["KV cache managers<br/>registered when built, held weakly<br/>v2 StorageStatistics, v1 allocated bytes"] S3["Loaded model<br/>claimed weakly; live parameters and<br/>buffers matched to allocator blocks"] S4["Startup steps and regions (full mode)<br/>allocation tags per executor step;<br/>NVML deltas for LoRA,<br/>KV transfer, torch.distributed NCCL"] S5["C++ native ledger<br/>recorded at alloc/free: NCCL comms,<br/>NVLS GEMM+AR, MoE profiler, Ulysses,<br/>EPLB, userbuffers"] S6["NCCL window allocator<br/>pooled window buffers"] S7["Component registries<br/>IPC buffers (custom AR, TP exchange),<br/>MNNVL, multicast AR, SymmMem, DeepEP,<br/>DWDP pages and local weight shards"] S8["NVML<br/>this process's total, by GPU UUID"] end M(["Memory monitor<br/>each byte in exactly one group;<br/>overhead = NVML total minus all others,<br/>so the groups sum to the process total"]) subgraph OUT["Report groups"] direction TB G1["KV cache<br/><i>pool capacity per pool group (main, draft, cross),<br/>KV auxiliary tensors; free blocks included;<br/>device-backed pools only, host-backed listed apart</i>"] G2["Model weights<br/><i>parameters and buffers, DWDP pages and<br/>local shards, EPLB expert slots</i>"] G3["Runtime buffers<br/><i>tagged startup allocations: model, sampler,<br/>spec decoding, executor; C++ workspaces</i>"] G4["CUDA graphs<br/><i>graph private pools, live and free;<br/>free space only the graph can reuse</i>"] G5["Communication buffers<br/><i>NCCL comms and windows, IPC buffers, MNNVL,<br/>NVLS/multicast, SymmMem, DeepEP, Ulysses,<br/>userbuffers</i>"] G6["PyTorch, unattributed<br/><i>live with no known owner: activations, leaks;<br/>split by allocating code with record_stacks</i>"] G7["PyTorch cache (free)<br/><i>reserved by PyTorch, not in use, reusable</i>"] G8["CUDA and library overhead<br/><i>CUDA context, cuBLAS, NCCL internals, loaded code,<br/>anything unmeasured; split by when it appeared</i>"] end S1 & S2 & S3 & S4 & S5 & S6 & S7 & S8 --> M M --> G1 & G2 & G3 & G4 & G5 & G6 & G7 & G8A real out-of-memory report (
memory_monitor_config=MemoryMonitorConfig(mode="full"), no environment variables; Llama-3.1-8B-Instruct, TP=2 on B200, rank 1). After a normal request, a logits processor leaked 4 GiB per step on every rank and then asked for more than was free. Both ranks reported; the request failed withRequestErrorand serving continued with unchanged outputs:The 12.00 GiB is the leak (3 x 4 GiB), and the suggestion separates it from what the profiled forward pass needs. In the default mode the same report has every row and the same total, but no "since warmup" column, no startup plan and no per-row breakdown.
Groups add up exactly. Every counted entry belongs to one group named for its purpose. The groups sum to the process total, and a test fails if any category has no group.
Growth is measured from warmup. Changes are shown against the report taken when startup completes, since every static allocation exists by then. A failed startup does not become the baseline.
Advice comes from the executor. Suggestions are the executor's own text, reused from
_ExecutorMemoryMonitor, and appear only for an OOM, including a startup that ran out of memory.It compares against the startup plan. KV cache sizing records its plan (the peak outside the KV cache), and current usage is compared with it. Free PyTorch cache is excluded from that comparison: sizing measured after emptying the cache, and the workload can reuse that space.
Startup steps are measured per process. Each executor creation step is measured through NVML. The executor's own figures are device-wide; on a shared GPU, two runs of the same model reported the weight-loading step as 1.08 GiB and 0.06 GiB.
Optional allocation stacks. With
TLLM_MEM_MONITOR_STACKS=1(off by default) the "PyTorch, unattributed" row is broken down by the code that allocated each block: the innermost TensorRT-LLM frame, or the third-party library. It records allocator state rather than an event history. Measured on torch 2.14, allocation got 26% slower and a snapshot took 6 ms; keeping the full history instead costs 47% and 2.3 s.Sleep mode is accounted for. Once
release_with_taghas run, torch still counts the released pools while the GPU no longer backs them. The monitor leaves released pools, and a released KV cache, out of the PyTorch footprint and out of every row, so the rows show only resident memory and still add up; one line lists what is released and returns on resume. A released tag counts only for the monitored executor's own sleep-mode memory, so another executor's sleep does not hide its resident pools. On a real release of the weights and model pools the process dropped from 19.63 GiB to 18.65 GiB. The overhead row stayed at 1.00 GiB; without the fix it would have fallen to about 0.Reports while serving. With
TLLM_MEM_MONITOR=1andTLLM_MEM_MONITOR_INTERVAL=<seconds>, a daemon thread reports at that interval once startup is over, so memory that grows slowly is visible before it runs out.Reports on request.
LLM.get_memory_report()(prototype API, also onAsyncLLM) returns one entry per rank, rank 0 first, in every mode: a report, or a line saying why that rank has none.MPI_THREAD_MULTIPLE; the KV-sizing pass gets none). A listener thread on each rank polls for requests and replies without blocking; a rank that does not answer within 30 s is reported as such. In the default mode the other ranks are listed as not gathered.trtllm-servewithmemory_monitor_config.mode: full(orTLLM_MEM_MONITOR=1), a runtime-control key (TRTLLM_RUNTIME_CONTROL_API_KEYor the server'sruntime_control_api_key) and a single serving frontend also servesGET /memory_report, with the runtime-control request signing like/memory_status; sleep mode and the runtime-control endpoints are not needed. Otherwise the route is not served and the server logs why. It is signed because a report exposes PIDs, other processes' usage and, in stack mode, source paths. Concurrent requests share one gather, and on-demand reports are not written to files. Weights are always claimed (a weak reference only), so a report names them in every mode.OOMs from any thread, reported once they are failures. torch's OOM observer only records the failed allocation (size, free memory, what PyTorch held); it builds no report, because callers such as warmup, KV sizing and optional workspaces catch OOMs and recover. The report is built when the error escapes. In every mode but
off, the executor's own error exits report it:_handle_errors(after the error responses are sent), a crashed_event_loop_wrapper(after its crash watchdog is armed; the watchdog now starts in theexceptblock), andvirtual_memory.materialize_with_tag, which reports every wakeup failure whoever called it. In full mode an ERROR log record (built on its own short-lived thread; background threads such as the KV transfer workers log their failures) andthreading.excepthookreport it too, and torch's OOM observer, installed only in full mode, puts the failed allocation into the headline. Errors are classified by exception type where one is at hand (tensorrt_llm._startup_errors.out_of_memory_kind, shared with startup error classification and the executor's OOM explanation), by text otherwise: host-memory failures (cudaHostRegister,cuMemHostAlloc, pinned memory,HostOOMError) never count, and failures that do not say which memory ran out (CUDA's genericout of memory,CuOOMError, NCCL'sunhandled cuda error) count only on a nearly full device. Reports are built on the executor's device, from any thread.Allocations by worker threads during startup. Tags are per thread (measured), so each tag's time window is also kept. An untagged allocation event inside a window takes the innermost window's label; explicit tags and claimed modules still take precedence.
Suggestions use growth since warmup. Warmup already runs the largest batch, so PyTorch memory with no known owner that grows past it means something is holding memory. The suggestion says so: it names the allocating code if stacks were recorded, and otherwise suggests
TLLM_MEM_MONITOR_STACKS=1. Growth of 1 GiB or more is explained in periodic reports too.Reports can be kept as files. With
report_dir/TLLM_MEM_MONITOR_DIRset, every report except on-demand ones is also written to its own file, in the default mode as well as full mode, named by rank, PID, UTC time and tag; the directory and files are private to their owner, and each process keeps the newest 100 periodic files it wrote. Logs from several ranks interleave and are often cut short when a job dies.Below the summary is a section broken down by allocator, plus conservation checks that compare independent sources, so a failure means the sources disagree.
How memory is attributed
get_kv_cache_stats().allocated_bytes; on Qwen2-0.5B this matched the engine's estimate, 17.54 GiB.StorageStatistics. The managers register themselves where they are built and are held weakly, so the monitor never keeps a pool alive. Only device-backed pools count; with sparse KV cache offload, pools of pinned host memory mapped into the device are listed separately as host-backed and not counted.ExecutorMemoryTypestep tags what it allocates. On Qwen2-0.5B, torch memory nobody could name fell from 0.17 GiB to 0.03 GiB.MemPoolobject and is not mistaken for a graph pool.memory_reserved().MNNVLAllReduce's class-level workspaces, both active and retiredNCCLWindowAllocator::getDeviceUsage, a new accessor read throughtrtllm::get_nccl_window_usagecommon/nativeMemory.h) that allocation sites record into by label, read throughtrtllm::get_native_memory_usage. Its labels are:• NCCL communicators. NCCL reports no size, so the drop in free memory across
ncclCommInitRankis recorded, and a reading that comes out negative (memory freed elsewhere meanwhile) makes the size unknown rather than wrong; withNCCL_RUNTIME_CONNECT=0, that is where NCCL allocates its connection buffers.• The fused GEMM+allreduce NVLS workspace, exact size.
• IPC buffers (custom allreduce, low-precision allreduce, TP peer exchange), exact size.
• The MoE GEMM profiler workspace, exact size, held while autotuning.
• Async Ulysses slots, exact size.
• Userbuffers (the torch.compile allreduce buffers), exact registered size, kept until exit.
• EPLB host-accessible memory, the requested size, labelled by backing; only the GDRCopy path is summed, and the NUMA path keeps its pages until exit and records no release.
torch.distributed). Regions nest without double counting, and those of the temporary KV-sizing executor are dropped when it is torn down. The parts sum to the row exactly.At TP=2 on Llama-3.1-8B, "Communication buffers" reads NCCL communicators 0.94 GiB, NCCL window 0.56 GiB and the custom allreduce workspace 0.50 GiB per rank, all named in every mode.
Reporting is armed once the monitored executor has been created, and suspended while it is being created, since KV sizing, warmup and graph capture deliberately run allocations into OOM and catch the error. The monitor follows one executor per process: a later executor in the same process is not monitored (no settings change, arming, startup record or providers, and its
get_memory_report()says why) and does not suspend reporting. Shutdown releases everything the monitor holds for the executor through one call,providers.end_executor(token), also when the executor loop ends withoutshutdown()or an earlier shutdown step raises.PagePoolalso gainsheld_pages(), plus a process-wide count of bytes whose release failed (unreleased_page_bytes()).Not in this PR
libtensorrt_llm, so it cannot use the ledger. It is created with the transceiver and is measured inside the "KV cache transfer buffers" region.TLLM_MEM_MONITOR_STACKS=1.torch.distributedconnects lazily after its first collective, and JIT-loaded module code, land in the overhead row, split by the startup step in which they appeared. Communicator memory is a device-wide measurement, so a concurrent allocation by another process on the same GPU would be counted too.get_rank_model_storage(pre-existing): it over-counts tied embeddings, reporting 1.17 GiB against 0.92 GiB actually held on Qwen2-0.5B.Follow-ups (not in this PR)
These came up in review and are deliberately left to follow-up PRs so that this one does not grow further:
LLM.get_memory_report()andGET /memory_reportreturn rendered text, one string per rank. A typed per-rank result would mirrorget_memory_status: rank, pid, device, status, totals, groups and checks, plus the text.MonitorCapturealready holds that structure. (review: discussion_r4551338999)describes(),collect_with_pool_rows()) onMonitorProviderwith defaults.hasattr.default_providers()reading other components' internals and the creator registering the KV pools._ExecutorMemoryMonitormeasures each executor creation step device-wide, and the monitor measures it per process. A single per-step record would let the startup-OOM explanation and the monitor share one measurement: per-process NVML readings in every mode, falling back to device-wide. (discussion_r4551339031)kv_budget_reserves. (discussion_r4551339042)VisualGen.get_memory_report()and atrtllm-serve --visual_gen_argsroute would be a public VisualGen API change, so they need the VisualGen team sync first. (discussion_r4551339010)TLLM_MEM_MONITOR_STACKS=1starts after a successful start. A warmup OOM (seen on DeepSeek-V4-Pro) can therefore be pinned to the step but not split into MoE scratch, attention buffers and activations. Add an opt-in mode that records stacks from model loading on, with its overhead documented (Python stacks alone slowed allocations by about 26%), and reconcile it with the startup-step tags.l0_gb300.yml, GB300-4_GPUs-PyTorch-1), so CI runs it first.Performance
Default (
oom). No added code runs per iteration, per forward or per allocation. Nothing is installed process-wide; the executor's error exits call the monitor when an error reaches them. Startup hooks cost about 1.6 µs each and take no allocator snapshot. The C++ ledger records only when native buffers are allocated or released. The process's memory pools are classified once when the executor has finished starting (one walk of the Python heap, under a second for 3M objects), so a report built during an OOM does not pay for it.Off (
0). As the default, without reporting.Full (
1). Arming installs PyTorch's OOM observer, a filter that acts only on ERROR log records, and athreading.excepthook; all three do nothing until something fails. Nothing runs per iteration either. The periodic reporter only wakes on its timer. Allocation history is recorded only during startup and is stopped when startup ends. Reports are cheap: one allocator snapshot each, and each pool is classified once. With 3M Python objects and a CUDA graph alive, the first report took 0.81 s, because it walks the heap once to classify pools, and later reports about 1 ms; before this change they took 0.14-0.18 s each.Throughput. Measured on Qwen2-0.5B, B200, 128 x 128 tokens, which is host-bound and so the most sensitive case. Ten interleaved runs each, pinned to idle cores: off averaged 24,100 tok/s and full 23,876 tok/s. The pairwise difference is -0.8% (range -5.5% to +3.5%), which is not distinguishable from noise. Against the base commit, which lacks this feature, the off runs measured 24,890-25,466 tok/s versus 24,939-25,192 tok/s. The default mode does a subset of the full mode's work and was not measured separately.
Stack mode.
TLLM_MEM_MONITOR_STACKS=1is the one mode with a steady-state cost: allocations are 26% slower in a microbenchmark, and the same run measured 24,675 tok/s. It is off by default and is documented as such.Measured at scale.
Token latency around a report (H100 SXM and H100 NVL; Llama-3.1-8B-Instruct, TP=2, CUDA graphs, 32 concurrent streaming requests, OSL 512;
get_memory_report()at 30/50/70 s; two rounds per mode). Across 24 calls the largest inter-token gap during a call was 13.7 ms, never above the largest gap the same run had without a call (12-15 ms; p99 5.7-7.4 ms). The first call is not slower than later ones, since pools are classified when the executor has finished starting. A report took 10-32 ms in the default mode and 113-264 ms in full mode, where rank 0 gathers the other rank's report on the caller's thread.Startup (H200 x8; each model four times on one node, off/full/full/off; compared on the warm-cache pair):
get_memory_report()at 70B TP8 took 0.17-0.45 s in full mode, gathering 8 ranks.Risk
By default the monitor installs nothing process-wide and only reports out-of-memory errors that reach the executor's error handling; mode
offturns even that off. Outside full mode, model loading takes no allocator snapshot and the tags touch no allocator state. Every provider, check and report is wrapped so that a failure is logged and skipped. The OOM handler allocates no device memory and swallows its own errors, so it can never replace theOutOfMemoryError.Outside the monitor package the change is a set of call sites:
model_loader.pypy_executor_creator.py_util.pypy_executor.py; also there, the sleep/wakeup ACK wait now uses the shared bounded receive intensorrt_llm/_torch/distributed/mpi_polling.py(_sleep_wakeup_ack_readyremoved); the monitor teardown runs fromshutdown()(in afinally) and at the end of_event_loop_wrapper; resource managers leave the monitor right before they are shut down; the crash watchdog in_event_loop_wrapperstarts in theexceptblock, before the error is reported and logged. No collective runs for the monitor: every rank reads the mode from the arguments.trtllm-serve: the/memory_reportroute, and/memory_statusnow shares its dispatch helper (_call_runtime_memory_method); the server readsmemory_monitor_configfrom the arguments and does not configure the monitor; the route snapshot gainsllm_memory_report, which needs API review committee sign-offLLM.get_memory_report/AsyncLLM.get_memory_report(prototype;tests/unittest/api_stability/references/llm.yamlupdated),BaseWorker.get_memory_report(an unmonitored engine answers with a "not monitored" line per rank), and the MPI multi-rank RPC allow-list inexecutor/proxy.pyvirtual_memory.py; pool registration inlocality_domain_utils.pydistributed/communicator.py,distributed/ops.py,_util.py(KV transfer) andresource_manager.py; exact IPC buffer sizes intensorrt_llm/_ipc_utils.pytensorrt_llm/_startup_errors.py:out_of_memory_kind, the shared OOM classifierpyexecutor/hang_detector.py:rank_crash_kill_pending(), so a failed startup skips its report while a crash kill is pending_torch/visual_gen/memory_monitor.pyand its hooks in_torch/visual_gen/executor.py(internal only; the public VisualGen API is unchanged)memory_monitor_configinllm_args.py(prototype; golden manifest and serve schemas regenerated), passed to the monitor by the executor creator andtrtllm-servecommon/nativeMemory.h/.cpp, its own static component so component-local gtests link) and its record calls, including one per registered userbuffer (kernels/userbuffers). These are a map update under a mutex on allocation and release only, never on the hot path. The label namespace isnativeMemoryLabels, andadjustNativeAllocationlowers a record's bytes without dropping its count. AlsoNCCLWindowAllocator::getDeviceUsage, which reads per-device totals under their own lock and never waits for an allocation or window registration, and three host-side torch ops (inthop/nativeMemoryOp.cpp).PagePoolaccessorsTest Coverage
The monitor tests run in CI.
tests/unittest/_torch/memoryhas 179 test functions, 228 test cases with a GPU; 192 of them arecpu_only, so the CPU stage runs them, andunittest/_torch/memoryis listed inl0_h100, whosenot cpu_onlyselection runs the rest. 50 more cases cover the executor hooks (tests/unittest/_torch/pyexecutor/test_executor_memory_monitor_hooks.py), VisualGen (tests/unittest/_torch/visual_gen/test_executor_memory_monitor.py), the serve route (tests/unittest/serve/test_memory_report_route.py), the MPI report (full_configanddefault) andmpi_polling. The page-pool test runs inl0_b200next to the other DWDP GPU tests, the two-rank MPI report test is collected by the existingunittest/_torch/multi_gpuentry, and the C++ ledger has a gtest (NativeMemoryTest). The Ulysses slot-ledger test is listed in the multi-GPUl0_dgx_b200block, the 2-GPU userbuffers ledger test runs with the existingtest_user_buffers.pyentry, and the EPLB native test is also listed in the GB300 pre-merge stage, where the allocator takes the NUMA path. Before this, none of the monitor's test files ran in CI: the CPU stage skips unmarked files, and no GPU stage listed the directory.User documentation:
docs/source/features/memory-monitor.md.228 monitor test cases in
tests/unittest/_torch/memoryplus the 50 above (tests that check one behaviour are merged, each case named in its assertion message), plus a 2-GPU Ulysses test and thetrtllm-serveroute tests. Real CUDA graph capture, the real sleep-mode pool with a realrelease_with_tag, real stack recording, a real MoE autotuning run and the real EPLB allocator are used where the behaviour depends on torch. Each rule that a measurement established has a test that fails when the rule is violated; this was verified by reverting each rule in turn. Some examples:test_graph_memory_is_cuda_graphs_not_reusable_cachetest_only_the_first_executor_in_a_process_is_monitoredtest_error_handler_reports_after_the_responsestest_freed_weights_do_not_label_their_successortest_manager_is_held_weaklytest_groups_add_up_and_every_category_has_a_grouptest_executor_memory_monitor_feeds_the_reporttest_check_c_can_fail_and_does_not_read_the_derived_propertiestest_pools_are_classified_by_kindtest_sleep_mode_pools_are_named_and_released_memory_leaves_the_footprinttest_sleep_mode_pools_are_named_and_released_memory_leaves_the_footprinttest_moe_gemm_profiler_workspace_is_recorded_while_profiling,test_eplb_host_accessible_memory_is_recordedtest_regions_count_each_byte_oncetest_regions_count_each_byte_oncetest_report_costtest_groups_add_up_when_allocator_counters_lag_the_snapshottest_startup_state_does_not_leakTLLM_MEM_MONITORis unsetTestDefaultReportsOnlyOutOfMemorytest_lora_device_cache_is_named,test_kv_transfer_buffers_are_named,test_sleep_listener_reports_a_failed_wakeup_oncetest_the_default_mode_installs_no_process_wide_hooktest_host_backed_pools_are_not_counted_as_device_memorytest_another_executors_sleep_does_not_release_the_monitored_kv_cachetest_a_loop_that_ends_without_shutdown_leaves_the_monitortest_stopping_frees_each_ranks_communicatorUsageIsReadWithoutWaitingForAnAllocationInProgresstest_a_block_with_frames_never_takes_a_frozen_startup_labeltest_other_errors_and_host_memory_are_not_reported,test_an_exception_is_classified_by_its_typeRegression on the touched files, at the head of this PR:
_torch/memory: 228 passed with a GPU, and 192cpu_onlypassed without one.test_py_executor.py,test_py_executor_creator_*, KV estimation,test_connector_adp,mpi_polling, startup errors, virtual memory, DWDP page pool and VisualGen: 855 passed.test_dynamic_eplb_contracts.py: 53 passed.tests/unittest/serve(cpu_only), the multi-frontend gate and the serve config schemas: 429 passed. The one failure,test_no_object_new, flags sites in other serve tests that this PR does not touch; it is waived (nvbugs/7081838).test_llm_args.pyandapi_stability: 709 passed.ctest -L common39/39 andNativeMemoryTest6/6.ncclUtilsTestwindow tests 45/45 per rank; the MPI report, userbuffers ledger and Ulysses tests 8 passed.End-to-end runs on Qwen2-0.5B-Instruct (B200) gave the following:
[checks] 3 passed, 0 FAILED.TLLM_MEM_MONITOR=1was not measurably slower than with the monitor off.OutOfMemoryErrorstill propagated.TLLM_MEM_MONITOR=1produced the same outputs before and after.TLLM_MEM_MONITOR_INTERVAL=5andTLLM_MEM_MONITOR_DIRwrote each rank's startup and periodic reports; rank 1's periodic reports name device 1.llm.get_memory_report()returned rank 0's report.TLLM_MEM_MONITOR=1,llm.get_memory_report()returned a report that names the model weights (0.92 GiB).Modes, on B200:
TLLM_MEM_MONITORunset,oom,0and1gave one OOM report, one, none, and one plus periodic reports.1, each rank wrote a startup report and an OOM report. Serving continued in both.Real runs after rebasing on main, with each report checked for exact sums and passing checks:
TLLM_MEM_MONITOR=1, KV cache V1, a real serving OOM, stack mode, sleep and wake, a startup that fails beside a colocated process, TP=2, a rank filter). Also Nemotron-H-8B (hybrid Mamba), andtrtllm-serveGET /memory_report(HTTP 200).The fused GEMM+allreduce NVLS record call is compiled but not exercised. Its NVFP4 model needs an FP8-KV attention kernel that
FAST_BUILDomits, and the JIT fallback for it cannot findcuda.hin this environment.PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code