Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
193 changes: 98 additions & 95 deletions cpp/tensorrt_llm/common/opUtils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@
#include <functional>
#include <mutex>
#include <thread>
#include <tuple>
#include <vector>

TRTLLM_NAMESPACE_BEGIN
#if ENABLE_MULTI_DEVICE
Expand Down Expand Up @@ -255,96 +257,109 @@ inline CUcontext getCurrentCudaCtx()
return ctx;
}

// Helper to create per-cuda-context and per-thread singleton managed by std::shared_ptr.
// Unlike conventional singletons, singleton created with this will be released
// when not needed, instead of on process exit.
// Objects of this class shall always be declared static / global, and shall never own CUDA
// resources.
//! Pool of cuBLAS / cuBLASLt handles grouped by CUDA context, with the same rule as PyTorch's DeviceThreadHandlePool:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Required] This is a private copy of PyTorch's DeviceThreadHandlePool, and PyTorch already pools per-thread cuBLAS/cuBLASLt handles for these callers: every caller of getCublasHandle()/getCublasLtHandle() is a thop op in th_common, which links torch, and the same thread holds PyTorch's pooled handle for its F.linear calls. at::cuda::getCurrentCUDABlasHandle()/getCurrentCUDABlasLtHandle() also leak the pool and take the cuBLAS workspace from the caching allocator, so KV-cache sizing sees it; the repo already moved DeepGEMM to the PyTorch-owned cuBLASLt handle (DG_USE_PYTORCH_CUBLASLT_HANDLE in cpp/tensorrt_llm/deep_gemm/CMakeLists.txt). With two pools each thread keeps two cuBLAS handles per GPU, each with a driver-allocated default workspace, and the pools differ in key (device vs CUcontext) and reset rule (PyTorch re-applies stream/workspace/math mode on every get), so later fixes have to be made twice. Please either pass PyTorch's handles to CublasMMWrapper from the thop callers (non-owning; its FP32 cublasGemmEx/stridedBatchedGemm paths would then set the math mode explicitly, since PyTorch applies allow_tf32), or say in the description why a separate pool stays and track the switch as a follow-up.

//! - A thread gets the handle it already holds in the current CUDA context; otherwise an idle handle of that context;
//! otherwise a new one.
//! - When a thread lets go of its handle (the last std::shared_ptr is released, usually when the thread exits), the
//! handle is not destroyed but goes back to the idle list of its CUDA context.
//! So a thread that starts late, when the PyTorch caching allocator may hold nearly all device memory, takes over a
//! handle instead of calling cublasCreate, which allocates its default workspace directly from the driver.
//!
//! Lifetime:
//! - Idle handles are never destroyed, not even at process exit: CUDA may already be unloaded by then, and
//! cublasDestroy, which synchronizes the device, can crash or hang.
//! - Per CUDA context, their number never exceeds the peak number of threads that held a handle of that context at

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] Handles return to the idle list when the last shared_ptr copy is released, not when a thread exits, so this bound holds only because every caller keeps the pointer in a thread_local or a local. Suggest: "their number is the peak number of handles held at the same time; a handle stays held until the last copy of its std::shared_ptr is released, even after its thread exits".

//! the same time.
//! - A context that ever had a handle must stay alive until process exit: otherwise its idle handles would dangle,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] This precondition is only documented: both tables are keyed by the CUcontext address and idle handles are never dropped, so a caller that uses these handles in a context it later destroys (a green context, cuCtxDestroy) would hand dead handles to a later context at the same address. No product path does that today. Keying by cuCtxGetId() (unique per process; the trtllm-gen export headers already key per-context caches on it, e.g. GemmInterface.h) removes the precondition: a destroyed context's idle handles are then just leaked.

//! and a context created later at the same address would be given them.
//! - The pool is a leaked singleton (hence the deleted destructor): a handle may still be given back after static
//! destruction has started (by a static object or by a thread still running at exit), and giveBack() needs a live
//! pool.
template <typename T>
class PerCudaCtxPerThreadSingletonCreator
class PerCudaCtxHandlePool

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] With the pool in this file's anonymous namespace, the only way to test it is through getCublasHandle() and process-global state: FULL_STACK, a GPU, and device-wide cudaMemGetInfo as the "no allocation" signal. A header with injectable create/reset functors would allow a host-only test with fake handles: count creates instead of probing memory, and drive acquire/giveBack (including the cross-thread release at lines 344-352) concurrently and deterministically.

{
public:
using CreatorFunc = std::function<std::unique_ptr<T>()>;
using DeleterFunc = std::function<void(T*)>;

// creator returning std::unique_ptr is by design.
// It forces separation of memory for T and memory for control blocks.
// So when T is released, but we still have observer weak_ptr in mObservers, the T mem block can be released.
// creator itself must not own CUDA resources. Only the object it creates can.
PerCudaCtxPerThreadSingletonCreator(CreatorFunc creator, DeleterFunc deleter)
: mCreator{std::move(creator)}
, mDeleter{std::move(deleter)}
, mObservers{std::make_unique<CacheType>()}
using CreateFunc = std::function<std::unique_ptr<T>()>;
using ResetFunc = std::function<void(T)>;

//! \param create Creates a handle in the current CUDA context.
//! \param resetForNewThread Clears the state that the previous thread may have left on an idle handle.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] The reset also runs when the same thread drops its last copy and asks again (a thread's first NVFP4 cuBLASLt GEMM does this: getOrCreateAlgoCache takes and releases the handle through a local wrapper, then cublas_fp4_gemm_caller_with_algo asks again). Suggest resetForReuse/mResetForReuse, documented as restoring what "a previous holder" left (same wording at line 427).

PerCudaCtxHandlePool(CreateFunc create, ResetFunc resetForNewThread)
: mCreate{std::move(create)}
, mResetForNewThread{std::move(resetForNewThread)}
{
}

~PerCudaCtxPerThreadSingletonCreator()
{
std::lock_guard<std::mutex> lk{mMutex};
mObservers.reset();
}
~PerCudaCtxHandlePool() = delete;

std::shared_ptr<T> operator()()
std::shared_ptr<T> acquire()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Required] CODING_GUIDELINES.md: "Follow Doxygen rules for documenting new class interfaces and function prototypes. For C++-style single-line comments use //!. For class members, use //!<." acquire() has no doc comment, takeIdleHandle()/giveBack() use // (lines 326, 339), and the member comments at lines 357 and 360-361 use //. Please document acquire() with //! (what it returns; the handle goes back to the pool when the last copy is released) and switch the others to //! / //!<.

{
std::lock_guard<std::mutex> lk{mMutex};
CUcontext ctx{getCurrentCudaCtx()};
std::thread::id thread = std::this_thread::get_id();
auto const key = std::make_tuple(ctx, thread);
std::shared_ptr<T> result = (*mObservers)[key].lock();
if (result == nullptr)
CUcontext const ctx = getCurrentCudaCtx();
HolderKey const key{ctx, std::this_thread::get_id()};
std::unique_ptr<T> handle;
{
TLLM_LOG_TRACE("creating singleton instance for CUDA context %lu and thread %lu", ctx, thread);
// Create the resource and register with an observer.
result = std::shared_ptr<T>{mCreator().release(),
[this, key](T* obj)
{
if (obj == nullptr)
{
return;
}
mDeleter(obj);

if (mObservers == nullptr)
{
return;
}

// Clears observer to avoid growth of mObservers, in case users creates/destroys cuda contexts
// frequently.
std::shared_ptr<T> observedObjHolder; // Delay destroy to avoid dead lock.
std::lock_guard<std::mutex> lk{mMutex};
// Must check observer again because another thread may created new instance for this ctx and this
// thread just before we lock mMutex. We can't infer that the observer is stale from the fact that
// obj is destroyed, because shared_ptr ref-count checking and observer removing are not in one
// atomic operation, and the observer may be changed to observe another instance.
auto it = mObservers->find(key);
if (it == mObservers->end())
{
return;
}
observedObjHolder = it->second.lock();
if (observedObjHolder == nullptr)
{
mObservers->erase(it);
}
}};
(*mObservers)[key] = result;
std::lock_guard<std::mutex> lk{mMutex};
if (auto held = mHeldHandles[key].lock())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] operator[] inserts an empty weak_ptr on a miss; if mCreate() or the reset then throws, the entry stays (against "mHeldHandles does not grow with exited threads") and a handle already taken from the idle list is lost with its box. Suggest find() here, and putting the handle back on the idle list if the reset throws.

{
return held;
}
handle = takeIdleHandle(ctx);
}
// Outside mMutex: resetting or creating a handle calls into CUDA.
if (handle)
{
mResetForNewThread(*handle);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] Taking over an idle handle, the path this fix relies on, logs nothing (the old TRACE lines are gone; only creation logs). Suggest TLLM_LOG_DEBUG("Reusing an idle handle in CUDA context %p", ctx); here, so a recurrence of Status: 3 shows which threads reused and which created a handle.

}
else
{
TLLM_LOG_TRACE("singleton instance for CUDA context %d and thread %d is cached", ctx, thread);
handle = mCreate();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Required] The pool only helps when an exited thread left an idle handle in the context. Flows that skip the KV-cache estimation run (TRTLLM_SKIP_KV_CACHE_ESTIMATION=1, which a deployment guide uses; the VANILLA backend; encoder-decoder models; CP/HELIX; sparse-KV-offload decode-only workers) have no profiling thread, so the serving thread's first TRT-LLM cuBLAS op still reaches this mCreate() -> cublasCreate() after the KV cache (fraction x free) and the warmup-filled torch cache have taken the memory, and can still fail with Failed to create cublas handle. Status: 3. That is unchanged from main, so not a regression, but the class comment (lines 265-266: "a thread that starts late ... takes over a handle instead of calling cublasCreate") and the description present this as a general fix for late-starting threads. Please state the scope (flows where an exited thread left a handle, i.e. the estimation flow), or make the serving path independent of thread history, e.g. acquire the worker's handles before the final KV cache is sized, or create the non-Lt handle lazily (see the comment on line 409).

}
std::shared_ptr<T> result{handle.release(), [this, key](T* released) { giveBack(key, released); }};
std::lock_guard<std::mutex> lk{mMutex};
mHeldHandles[key] = result;
return result;
}

private:
CreatorFunc mCreator;
DeleterFunc mDeleter;
mutable std::mutex mMutex;
// CUDA resources are per-context and per-thread.
using CacheKey = std::tuple<CUcontext, std::thread::id>;
using CacheType = std::unordered_map<CacheKey, std::weak_ptr<T>, common::op::OpCustomHash<CacheKey>>;
std::unique_ptr<CacheType> mObservers;
using HolderKey = std::tuple<CUcontext, std::thread::id>;

// Requires mMutex.
std::unique_ptr<T> takeIdleHandle(CUcontext ctx)
{
auto& idleHandles = mIdleHandles[ctx];
if (idleHandles.empty())
{
return nullptr;
}
auto handle = std::move(idleHandles.back());
idleHandles.pop_back();
return handle;
}

// Deleter of the std::shared_ptr returned by acquire().
void giveBack(HolderKey const& key, T* handle)
{
std::lock_guard<std::mutex> lk{mMutex};
mIdleHandles[std::get<CUcontext>(key)].emplace_back(handle);
// Forget the holder so that mHeldHandles does not grow with exited threads. Keep the entry if it already
// observes a newer handle: when the last owner was another thread, the holder may have acquired again before
// this call took mMutex. expired() rather than lock(): a std::shared_ptr taken here could become the last
// owner and re-enter giveBack() while mMutex is held.
auto const it = mHeldHandles.find(key);
if (it != mHeldHandles.end() && it->second.expired())
{
mHeldHandles.erase(it);
}
}

CreateFunc mCreate;
ResetFunc mResetForNewThread;
// Guards only the two tables below; no CUDA call is made while holding it.
std::mutex mMutex;
std::unordered_map<HolderKey, std::weak_ptr<T>, common::op::OpCustomHash<HolderKey>> mHeldHandles;
// Each std::unique_ptr owns only the box that holds a handle value: freeing a box leaks the handle, it never
// destroys it.
std::unordered_map<CUcontext, std::vector<std::unique_ptr<T>>> mIdleHandles;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] Storing unique_ptr boxes in the idle list is what needs the "freeing a box leaks the handle" warning. Storing values (std::vector<T>), freeing the box in giveBack() and allocating it in acquire() removes the extra allocation per idle handle and the trap.

};

// Structure to hold memory information
Expand Down Expand Up @@ -391,7 +406,7 @@ void throwCublasErrorWithMemInfo(char const* operation, CUcontext ctx, cublasSta

std::shared_ptr<cublasHandle_t> getCublasHandle()
{
static PerCudaCtxPerThreadSingletonCreator<cublasHandle_t> creator(
static auto* const pool = new PerCudaCtxHandlePool<cublasHandle_t>(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] The handle whose cublasCreate failed in this bug is one the failing op never uses: CublasMMWrapper takes both handles up front, but trtllm::cublas_mm/cublas_scaled_mm, the FP4 runner, FP16/BF16/FP8 LoRA and fused attention only call cuBLASLt; the cuBLAS handle is used only by the FP32 Gemm branch and stridedBatchedGemm (MHA's unfused path). In the DeepSeek-V3-Lite repro, the serving thread's consumer is kv_a_proj_with_mqa's BF16 trtllm::cublas_mm. Acquiring the cuBLAS handle lazily on the first non-Lt call (and moving the cublasSetStream at attentionOp.cpp:1020 into the unfused branch) would drop that ~64 MiB driver allocation on Lt-only paths whatever the thread history. Fine as a follow-up.

[]() -> auto
{
CUcontext ctx = getCurrentCudaCtx();
Expand All @@ -407,22 +422,18 @@ std::shared_ptr<cublasHandle_t> getCublasHandle()

return handle;
},
[](cublasHandle_t* handle)
[](cublasHandle_t handle)
{
auto status = cublasDestroy(*handle);
if (status != CUBLAS_STATUS_SUCCESS)
{
TLLM_LOG_WARNING("Failed to destroy cublas handle. Status: %d", status);
}
delete handle;
handle = nullptr;
// The previous thread may have left its stream and workspace on the handle (CublasMMWrapper sets both),
// and both may be gone with it. cublasSetStream also resets the workspace to the default one.
TLLM_CUDA_CHECK(cublasSetStream(handle, nullptr));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] This restores only the stream and workspace; math mode, pointer mode, atomics mode and the SM-count target also travel with the handle, while the constructor comment says the reset clears "the state that the previous thread may have left". No caller sets them today, but a later TF32 math mode or device pointer mode would leak into whichever thread reuses the handle. Either also reset them (cublasSetMathMode(handle, CUBLAS_DEFAULT_MATH), cublasSetPointerMode(handle, CUBLAS_POINTER_MODE_HOST)) or narrow the comment to stream and workspace.

});
return creator();
return pool->acquire();
}

std::shared_ptr<cublasLtHandle_t> getCublasLtHandle()
{
static PerCudaCtxPerThreadSingletonCreator<cublasLtHandle_t> creator(
static auto* const pool = new PerCudaCtxHandlePool<cublasLtHandle_t>(
[]() -> auto
{
CUcontext ctx = getCurrentCudaCtx();
Expand All @@ -438,17 +449,9 @@ std::shared_ptr<cublasLtHandle_t> getCublasLtHandle()

return handle;
},
[](cublasLtHandle_t* handle)
{
auto status = cublasLtDestroy(*handle);
if (status != CUBLAS_STATUS_SUCCESS)
{
TLLM_LOG_WARNING("Failed to destroy cublasLt handle. Status: %d", status);
}
delete handle;
handle = nullptr;
});
return creator();
// cuBLASLt takes the stream and workspace with each call, so a handle keeps no state of the previous thread.
[](cublasLtHandle_t) {});
return pool->acquire();
}

TRTLLM_NAMESPACE_END
5 changes: 3 additions & 2 deletions cpp/tensorrt_llm/common/opUtils.h
Original file line number Diff line number Diff line change
Expand Up @@ -156,8 +156,9 @@ std::shared_ptr<ncclComm_t> getComm(std::set<int> const& group);

#endif // ENABLE_MULTI_DEVICE

//! To save GPU memory, all the plugins share the same cublas and cublasLt handle globally.
//! Get cublas and cublasLt handle for current cuda context
//! Get the cuBLAS / cuBLASLt handle of the calling thread in the current CUDA context. When the last copy of the
//! returned std::shared_ptr is released, the handle is kept for the next thread that asks in the same CUDA context

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] This is the API doc of both getters, and it leaves out the new contract: handles are never destroyed (not even at process exit), the thread that released a handle can get it back reset too, and a CUDA context in which a handle was used must not be destroyed. Suggest: //! ... the handle is not destroyed, not even at process exit, but kept for the next request in the same CUDA context, from any thread; a cuBLAS handle is reset to the default stream and workspace before it is reused. A CUDA context in which a handle was requested must not be destroyed before process exit.

//! instead of being destroyed; a cuBLAS handle is reset to the default stream and workspace before it is reused.
std::shared_ptr<cublasHandle_t> getCublasHandle();
std::shared_ptr<cublasLtHandle_t> getCublasLtHandle();

Expand Down
9 changes: 9 additions & 0 deletions cpp/tests/unit_tests/common/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,15 @@ add_tllm_gtest(
"unit;common;gpu")
add_tllm_gtest(attentionWorkspaceTest SOURCES attentionWorkspaceTest.cpp
FULL_STACK LABELS "unit;full-stack;gpu")
add_tllm_gtest(
cublasHandlePoolTest
OWNER
common
SOURCES
cublasHandlePoolTest.cpp
FULL_STACK
LABELS
"unit;full-stack;gpu")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Required] cublasHandlePoolTest has OWNER common but its LABELS lack the component label common. AGENTS.md asks new C++ unit tests to use add_tllm_gtest with explicit OWNER and scheduling LABELS, and documents ctest --test-dir cpp/build_RelWithDebInfo -L common for scheduling the common tests: that command never selects this test, while --target common-tests now has to link the whole libtensorrt_llm for it (every other common-owned test is component-local). FULL_STACK itself is needed, since opUtils.cpp only lives in the common_src object library, and CI runs the directory without -L, so CI is not affected. Fix: LABELS "unit;common;full-stack;gpu" (as fmhaHead104Test carries kernels), or drop OWNER common like attentionWorkspaceTest if common-tests should stay component-local.

add_tllm_gtest(
memoryUtilsTest
OWNER
Expand Down
103 changes: 103 additions & 0 deletions cpp/tests/unit_tests/common/cublasHandlePoolTest.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/*
* SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
* SPDX-License-Identifier: Apache-2.0
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

#include "tensorrt_llm/common/cudaUtils.h"
#include "tensorrt_llm/common/opUtils.h"

#include <gtest/gtest.h>

#include <cstddef>
#include <thread>
#include <utility>

namespace
{

size_t getFreeDeviceMemory()
{
size_t freeBytes = 0;
size_t totalBytes = 0;
TLLM_CUDA_CHECK(cudaMemGetInfo(&freeBytes, &totalBytes));
return freeBytes;
}

template <typename Func>
void runInNewThread(Func&& func)
{
std::thread{std::forward<Func>(func)}.join();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] An exception from TLLM_CUDA_CHECK or getCublasHandle() inside the thread calls std::terminate, so a failure shows up as an aborted process rather than a gtest failure. std::async(std::launch::async, std::forward<Func>(func)).get(); rethrows on the test thread.

}

} // namespace

// Only cuBLAS handles are checked: cublasLtCreate returns the same handle on every call (cuBLAS 13), so how cuBLASLt

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] Skipping cuBLASLt rests on a version-specific behavior; after a cuBLAS change the Lt take-over path would become observable but stay untested. Suggest a runtime check: if two live threads get different Lt handles, run the same take-over assertions, else GTEST_SKIP() with this reason; and a line in getCublasLtHandle() on why it is pooled.

// handles are handed out is not observable.

// A thread started late (such as the serving thread that replaces the profiling thread) may find the PyTorch caching
// allocator holding nearly all device memory, so it must take over the handle of an exited thread instead of allocating
// a new one. The exited thread's stream and workspace may be gone with it, so the taken-over handle must be back on the
// default stream (cuBLAS has no getter for the workspace; cublasSetStream resets it together with the stream).
TEST(CublasHandlePoolTest, NewThreadTakesOverCublasHandleOfExitedThreadOnDefaultStreamWithoutAllocating)
{
constexpr size_t kWorkspaceSize = 1 << 20;
cudaStream_t streamOfExitedThread{};
void* workspaceOfExitedThread{};
TLLM_CUDA_CHECK(cudaStreamCreate(&streamOfExitedThread));
TLLM_CUDA_CHECK(cudaMalloc(&workspaceOfExitedThread, kWorkspaceSize));
cublasHandle_t handleOfExitedThread{};
runInNewThread(
[&]
{
auto const handle = tensorrt_llm::getCublasHandle();
handleOfExitedThread = *handle;
// As CublasMMWrapper does.
TLLM_CUDA_CHECK(cublasSetStream(*handle, streamOfExitedThread));
TLLM_CUDA_CHECK(cublasSetWorkspace(*handle, workspaceOfExitedThread, kWorkspaceSize));
});
TLLM_CUDA_CHECK(cudaFree(workspaceOfExitedThread));
TLLM_CUDA_CHECK(cudaStreamDestroy(streamOfExitedThread));

cublasHandle_t handleOfNewThread{};
cudaStream_t streamOfNewThread{};
size_t freeMemoryBefore = 0;
size_t freeMemoryAfter = 0;
runInNewThread(
[&]
{
freeMemoryBefore = getFreeDeviceMemory();
auto const handle = tensorrt_llm::getCublasHandle();
freeMemoryAfter = getFreeDeviceMemory();
handleOfNewThread = *handle;
TLLM_CUDA_CHECK(cublasGetStream(*handle, &streamOfNewThread));
});

ASSERT_EQ(handleOfNewThread, handleOfExitedThread);
EXPECT_EQ(streamOfNewThread, nullptr);
EXPECT_EQ(freeMemoryAfter, freeMemoryBefore);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] cudaMemGetInfo is device-wide, and CI runs this directory with CTEST_PARALLEL_LEVEL=2, each TEST in its own process (the other TEST here calls cublasCreate, ~64 MiB). A concurrent allocation or free between lines 80 and 82 fails this check; the serial --rerun-failed hides it in CI, a local ctest -j does not. Since this is the only assertion that reliably fails on main (line 87 can pass if the new handle reuses the freed address), please harden rather than loosen it: e.g. repeat measure/acquire/measure in fresh threads and pass if any attempt shows no change.

}

TEST(CublasHandlePoolTest, LiveThreadsGetDistinctCublasHandlesAndEachThreadKeepsItsOwn)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nice to have] Both tests are sequential with at most two live handles, so acquire()/giveBack() never run under contention, and nothing else in CI calls getCublasHandle() concurrently. Suggest N threads x a few rounds (acquire, barrier, record, release), asserting distinct handles within a round and at most N handles over all rounds.

{
// Leave one idle handle: the main thread takes it, so the other thread must get a different one.
runInNewThread([] { tensorrt_llm::getCublasHandle(); });
auto const handle = tensorrt_llm::getCublasHandle();
EXPECT_EQ(*tensorrt_llm::getCublasHandle(), *handle);

cublasHandle_t handleOfOtherThread{};
runInNewThread([&] { handleOfOtherThread = *tensorrt_llm::getCublasHandle(); });

EXPECT_NE(handleOfOtherThread, *handle);
}
Loading