Repository navigation
[https://nvbugs/7063452][fix] Reuse cuBLAS handles of exited threads #22186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,6 +31,8 @@ | |
| #include <functional> | ||
| #include <mutex> | ||
| #include <thread> | ||
| #include <tuple> | ||
| #include <vector> | ||
|
|
||
| TRTLLM_NAMESPACE_BEGIN | ||
| #if ENABLE_MULTI_DEVICE | ||
|
|
@@ -255,96 +257,109 @@ inline CUcontext getCurrentCudaCtx() | |
| return ctx; | ||
| } | ||
|
|
||
| // Helper to create per-cuda-context and per-thread singleton managed by std::shared_ptr. | ||
| // Unlike conventional singletons, singleton created with this will be released | ||
| // when not needed, instead of on process exit. | ||
| // Objects of this class shall always be declared static / global, and shall never own CUDA | ||
| // resources. | ||
| //! Pool of cuBLAS / cuBLASLt handles grouped by CUDA context, with the same rule as PyTorch's DeviceThreadHandlePool: | ||
| //! - A thread gets the handle it already holds in the current CUDA context; otherwise an idle handle of that context; | ||
| //! otherwise a new one. | ||
| //! - When a thread lets go of its handle (the last std::shared_ptr is released, usually when the thread exits), the | ||
| //! handle is not destroyed but goes back to the idle list of its CUDA context. | ||
| //! So a thread that starts late, when the PyTorch caching allocator may hold nearly all device memory, takes over a | ||
| //! handle instead of calling cublasCreate, which allocates its default workspace directly from the driver. | ||
| //! | ||
| //! Lifetime: | ||
| //! - Idle handles are never destroyed, not even at process exit: CUDA may already be unloaded by then, and | ||
| //! cublasDestroy, which synchronizes the device, can crash or hang. | ||
| //! - Per CUDA context, their number never exceeds the peak number of threads that held a handle of that context at | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] Handles return to the idle list when the last |
||
| //! the same time. | ||
| //! - A context that ever had a handle must stay alive until process exit: otherwise its idle handles would dangle, | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] This precondition is only documented: both tables are keyed by the |
||
| //! and a context created later at the same address would be given them. | ||
| //! - The pool is a leaked singleton (hence the deleted destructor): a handle may still be given back after static | ||
| //! destruction has started (by a static object or by a thread still running at exit), and giveBack() needs a live | ||
| //! pool. | ||
| template <typename T> | ||
| class PerCudaCtxPerThreadSingletonCreator | ||
| class PerCudaCtxHandlePool | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] With the pool in this file's anonymous namespace, the only way to test it is through |
||
| { | ||
| public: | ||
| using CreatorFunc = std::function<std::unique_ptr<T>()>; | ||
| using DeleterFunc = std::function<void(T*)>; | ||
|
|
||
| // creator returning std::unique_ptr is by design. | ||
| // It forces separation of memory for T and memory for control blocks. | ||
| // So when T is released, but we still have observer weak_ptr in mObservers, the T mem block can be released. | ||
| // creator itself must not own CUDA resources. Only the object it creates can. | ||
| PerCudaCtxPerThreadSingletonCreator(CreatorFunc creator, DeleterFunc deleter) | ||
| : mCreator{std::move(creator)} | ||
| , mDeleter{std::move(deleter)} | ||
| , mObservers{std::make_unique<CacheType>()} | ||
| using CreateFunc = std::function<std::unique_ptr<T>()>; | ||
| using ResetFunc = std::function<void(T)>; | ||
|
|
||
| //! \param create Creates a handle in the current CUDA context. | ||
| //! \param resetForNewThread Clears the state that the previous thread may have left on an idle handle. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] The reset also runs when the same thread drops its last copy and asks again (a thread's first NVFP4 cuBLASLt GEMM does this: |
||
| PerCudaCtxHandlePool(CreateFunc create, ResetFunc resetForNewThread) | ||
| : mCreate{std::move(create)} | ||
| , mResetForNewThread{std::move(resetForNewThread)} | ||
| { | ||
| } | ||
|
|
||
| ~PerCudaCtxPerThreadSingletonCreator() | ||
| { | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| mObservers.reset(); | ||
| } | ||
| ~PerCudaCtxHandlePool() = delete; | ||
|
|
||
| std::shared_ptr<T> operator()() | ||
| std::shared_ptr<T> acquire() | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Required] CODING_GUIDELINES.md: "Follow Doxygen rules for documenting new class interfaces and function prototypes. For C++-style single-line comments use |
||
| { | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| CUcontext ctx{getCurrentCudaCtx()}; | ||
| std::thread::id thread = std::this_thread::get_id(); | ||
| auto const key = std::make_tuple(ctx, thread); | ||
| std::shared_ptr<T> result = (*mObservers)[key].lock(); | ||
| if (result == nullptr) | ||
| CUcontext const ctx = getCurrentCudaCtx(); | ||
| HolderKey const key{ctx, std::this_thread::get_id()}; | ||
| std::unique_ptr<T> handle; | ||
| { | ||
| TLLM_LOG_TRACE("creating singleton instance for CUDA context %lu and thread %lu", ctx, thread); | ||
| // Create the resource and register with an observer. | ||
| result = std::shared_ptr<T>{mCreator().release(), | ||
| [this, key](T* obj) | ||
| { | ||
| if (obj == nullptr) | ||
| { | ||
| return; | ||
| } | ||
| mDeleter(obj); | ||
|
|
||
| if (mObservers == nullptr) | ||
| { | ||
| return; | ||
| } | ||
|
|
||
| // Clears observer to avoid growth of mObservers, in case users creates/destroys cuda contexts | ||
| // frequently. | ||
| std::shared_ptr<T> observedObjHolder; // Delay destroy to avoid dead lock. | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| // Must check observer again because another thread may created new instance for this ctx and this | ||
| // thread just before we lock mMutex. We can't infer that the observer is stale from the fact that | ||
| // obj is destroyed, because shared_ptr ref-count checking and observer removing are not in one | ||
| // atomic operation, and the observer may be changed to observe another instance. | ||
| auto it = mObservers->find(key); | ||
| if (it == mObservers->end()) | ||
| { | ||
| return; | ||
| } | ||
| observedObjHolder = it->second.lock(); | ||
| if (observedObjHolder == nullptr) | ||
| { | ||
| mObservers->erase(it); | ||
| } | ||
| }}; | ||
| (*mObservers)[key] = result; | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| if (auto held = mHeldHandles[key].lock()) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] |
||
| { | ||
| return held; | ||
| } | ||
| handle = takeIdleHandle(ctx); | ||
| } | ||
| // Outside mMutex: resetting or creating a handle calls into CUDA. | ||
| if (handle) | ||
| { | ||
| mResetForNewThread(*handle); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] Taking over an idle handle, the path this fix relies on, logs nothing (the old TRACE lines are gone; only creation logs). Suggest |
||
| } | ||
| else | ||
| { | ||
| TLLM_LOG_TRACE("singleton instance for CUDA context %d and thread %d is cached", ctx, thread); | ||
| handle = mCreate(); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Required] The pool only helps when an exited thread left an idle handle in the context. Flows that skip the KV-cache estimation run ( |
||
| } | ||
| std::shared_ptr<T> result{handle.release(), [this, key](T* released) { giveBack(key, released); }}; | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| mHeldHandles[key] = result; | ||
| return result; | ||
| } | ||
|
|
||
| private: | ||
| CreatorFunc mCreator; | ||
| DeleterFunc mDeleter; | ||
| mutable std::mutex mMutex; | ||
| // CUDA resources are per-context and per-thread. | ||
| using CacheKey = std::tuple<CUcontext, std::thread::id>; | ||
| using CacheType = std::unordered_map<CacheKey, std::weak_ptr<T>, common::op::OpCustomHash<CacheKey>>; | ||
| std::unique_ptr<CacheType> mObservers; | ||
| using HolderKey = std::tuple<CUcontext, std::thread::id>; | ||
|
|
||
| // Requires mMutex. | ||
| std::unique_ptr<T> takeIdleHandle(CUcontext ctx) | ||
| { | ||
| auto& idleHandles = mIdleHandles[ctx]; | ||
| if (idleHandles.empty()) | ||
| { | ||
| return nullptr; | ||
| } | ||
| auto handle = std::move(idleHandles.back()); | ||
| idleHandles.pop_back(); | ||
| return handle; | ||
| } | ||
|
|
||
| // Deleter of the std::shared_ptr returned by acquire(). | ||
| void giveBack(HolderKey const& key, T* handle) | ||
| { | ||
| std::lock_guard<std::mutex> lk{mMutex}; | ||
| mIdleHandles[std::get<CUcontext>(key)].emplace_back(handle); | ||
| // Forget the holder so that mHeldHandles does not grow with exited threads. Keep the entry if it already | ||
| // observes a newer handle: when the last owner was another thread, the holder may have acquired again before | ||
| // this call took mMutex. expired() rather than lock(): a std::shared_ptr taken here could become the last | ||
| // owner and re-enter giveBack() while mMutex is held. | ||
| auto const it = mHeldHandles.find(key); | ||
| if (it != mHeldHandles.end() && it->second.expired()) | ||
| { | ||
| mHeldHandles.erase(it); | ||
| } | ||
| } | ||
|
|
||
| CreateFunc mCreate; | ||
| ResetFunc mResetForNewThread; | ||
| // Guards only the two tables below; no CUDA call is made while holding it. | ||
| std::mutex mMutex; | ||
| std::unordered_map<HolderKey, std::weak_ptr<T>, common::op::OpCustomHash<HolderKey>> mHeldHandles; | ||
| // Each std::unique_ptr owns only the box that holds a handle value: freeing a box leaks the handle, it never | ||
| // destroys it. | ||
| std::unordered_map<CUcontext, std::vector<std::unique_ptr<T>>> mIdleHandles; | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] Storing |
||
| }; | ||
|
|
||
| // Structure to hold memory information | ||
|
|
@@ -391,7 +406,7 @@ void throwCublasErrorWithMemInfo(char const* operation, CUcontext ctx, cublasSta | |
|
|
||
| std::shared_ptr<cublasHandle_t> getCublasHandle() | ||
| { | ||
| static PerCudaCtxPerThreadSingletonCreator<cublasHandle_t> creator( | ||
| static auto* const pool = new PerCudaCtxHandlePool<cublasHandle_t>( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] The handle whose |
||
| []() -> auto | ||
| { | ||
| CUcontext ctx = getCurrentCudaCtx(); | ||
|
|
@@ -407,22 +422,18 @@ std::shared_ptr<cublasHandle_t> getCublasHandle() | |
|
|
||
| return handle; | ||
| }, | ||
| [](cublasHandle_t* handle) | ||
| [](cublasHandle_t handle) | ||
| { | ||
| auto status = cublasDestroy(*handle); | ||
| if (status != CUBLAS_STATUS_SUCCESS) | ||
| { | ||
| TLLM_LOG_WARNING("Failed to destroy cublas handle. Status: %d", status); | ||
| } | ||
| delete handle; | ||
| handle = nullptr; | ||
| // The previous thread may have left its stream and workspace on the handle (CublasMMWrapper sets both), | ||
| // and both may be gone with it. cublasSetStream also resets the workspace to the default one. | ||
| TLLM_CUDA_CHECK(cublasSetStream(handle, nullptr)); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] This restores only the stream and workspace; math mode, pointer mode, atomics mode and the SM-count target also travel with the handle, while the constructor comment says the reset clears "the state that the previous thread may have left". No caller sets them today, but a later TF32 math mode or device pointer mode would leak into whichever thread reuses the handle. Either also reset them ( |
||
| }); | ||
| return creator(); | ||
| return pool->acquire(); | ||
| } | ||
|
|
||
| std::shared_ptr<cublasLtHandle_t> getCublasLtHandle() | ||
| { | ||
| static PerCudaCtxPerThreadSingletonCreator<cublasLtHandle_t> creator( | ||
| static auto* const pool = new PerCudaCtxHandlePool<cublasLtHandle_t>( | ||
| []() -> auto | ||
| { | ||
| CUcontext ctx = getCurrentCudaCtx(); | ||
|
|
@@ -438,17 +449,9 @@ std::shared_ptr<cublasLtHandle_t> getCublasLtHandle() | |
|
|
||
| return handle; | ||
| }, | ||
| [](cublasLtHandle_t* handle) | ||
| { | ||
| auto status = cublasLtDestroy(*handle); | ||
| if (status != CUBLAS_STATUS_SUCCESS) | ||
| { | ||
| TLLM_LOG_WARNING("Failed to destroy cublasLt handle. Status: %d", status); | ||
| } | ||
| delete handle; | ||
| handle = nullptr; | ||
| }); | ||
| return creator(); | ||
| // cuBLASLt takes the stream and workspace with each call, so a handle keeps no state of the previous thread. | ||
| [](cublasLtHandle_t) {}); | ||
| return pool->acquire(); | ||
| } | ||
|
|
||
| TRTLLM_NAMESPACE_END | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -156,8 +156,9 @@ std::shared_ptr<ncclComm_t> getComm(std::set<int> const& group); | |
|
|
||
| #endif // ENABLE_MULTI_DEVICE | ||
|
|
||
| //! To save GPU memory, all the plugins share the same cublas and cublasLt handle globally. | ||
| //! Get cublas and cublasLt handle for current cuda context | ||
| //! Get the cuBLAS / cuBLASLt handle of the calling thread in the current CUDA context. When the last copy of the | ||
| //! returned std::shared_ptr is released, the handle is kept for the next thread that asks in the same CUDA context | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] This is the API doc of both getters, and it leaves out the new contract: handles are never destroyed (not even at process exit), the thread that released a handle can get it back reset too, and a CUDA context in which a handle was used must not be destroyed. Suggest: |
||
| //! instead of being destroyed; a cuBLAS handle is reset to the default stream and workspace before it is reused. | ||
| std::shared_ptr<cublasHandle_t> getCublasHandle(); | ||
| std::shared_ptr<cublasLtHandle_t> getCublasLtHandle(); | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,6 +35,15 @@ add_tllm_gtest( | |
| "unit;common;gpu") | ||
| add_tllm_gtest(attentionWorkspaceTest SOURCES attentionWorkspaceTest.cpp | ||
| FULL_STACK LABELS "unit;full-stack;gpu") | ||
| add_tllm_gtest( | ||
| cublasHandlePoolTest | ||
| OWNER | ||
| common | ||
| SOURCES | ||
| cublasHandlePoolTest.cpp | ||
| FULL_STACK | ||
| LABELS | ||
| "unit;full-stack;gpu") | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Required] |
||
| add_tllm_gtest( | ||
| memoryUtilsTest | ||
| OWNER | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| /* | ||
| * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| * SPDX-License-Identifier: Apache-2.0 | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| #include "tensorrt_llm/common/cudaUtils.h" | ||
| #include "tensorrt_llm/common/opUtils.h" | ||
|
|
||
| #include <gtest/gtest.h> | ||
|
|
||
| #include <cstddef> | ||
| #include <thread> | ||
| #include <utility> | ||
|
|
||
| namespace | ||
| { | ||
|
|
||
| size_t getFreeDeviceMemory() | ||
| { | ||
| size_t freeBytes = 0; | ||
| size_t totalBytes = 0; | ||
| TLLM_CUDA_CHECK(cudaMemGetInfo(&freeBytes, &totalBytes)); | ||
| return freeBytes; | ||
| } | ||
|
|
||
| template <typename Func> | ||
| void runInNewThread(Func&& func) | ||
| { | ||
| std::thread{std::forward<Func>(func)}.join(); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] An exception from |
||
| } | ||
|
|
||
| } // namespace | ||
|
|
||
| // Only cuBLAS handles are checked: cublasLtCreate returns the same handle on every call (cuBLAS 13), so how cuBLASLt | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] Skipping cuBLASLt rests on a version-specific behavior; after a cuBLAS change the Lt take-over path would become observable but stay untested. Suggest a runtime check: if two live threads get different Lt handles, run the same take-over assertions, else |
||
| // handles are handed out is not observable. | ||
|
|
||
| // A thread started late (such as the serving thread that replaces the profiling thread) may find the PyTorch caching | ||
| // allocator holding nearly all device memory, so it must take over the handle of an exited thread instead of allocating | ||
| // a new one. The exited thread's stream and workspace may be gone with it, so the taken-over handle must be back on the | ||
| // default stream (cuBLAS has no getter for the workspace; cublasSetStream resets it together with the stream). | ||
| TEST(CublasHandlePoolTest, NewThreadTakesOverCublasHandleOfExitedThreadOnDefaultStreamWithoutAllocating) | ||
| { | ||
| constexpr size_t kWorkspaceSize = 1 << 20; | ||
| cudaStream_t streamOfExitedThread{}; | ||
| void* workspaceOfExitedThread{}; | ||
| TLLM_CUDA_CHECK(cudaStreamCreate(&streamOfExitedThread)); | ||
| TLLM_CUDA_CHECK(cudaMalloc(&workspaceOfExitedThread, kWorkspaceSize)); | ||
| cublasHandle_t handleOfExitedThread{}; | ||
| runInNewThread( | ||
| [&] | ||
| { | ||
| auto const handle = tensorrt_llm::getCublasHandle(); | ||
| handleOfExitedThread = *handle; | ||
| // As CublasMMWrapper does. | ||
| TLLM_CUDA_CHECK(cublasSetStream(*handle, streamOfExitedThread)); | ||
| TLLM_CUDA_CHECK(cublasSetWorkspace(*handle, workspaceOfExitedThread, kWorkspaceSize)); | ||
| }); | ||
| TLLM_CUDA_CHECK(cudaFree(workspaceOfExitedThread)); | ||
| TLLM_CUDA_CHECK(cudaStreamDestroy(streamOfExitedThread)); | ||
|
|
||
| cublasHandle_t handleOfNewThread{}; | ||
| cudaStream_t streamOfNewThread{}; | ||
| size_t freeMemoryBefore = 0; | ||
| size_t freeMemoryAfter = 0; | ||
| runInNewThread( | ||
| [&] | ||
| { | ||
| freeMemoryBefore = getFreeDeviceMemory(); | ||
| auto const handle = tensorrt_llm::getCublasHandle(); | ||
| freeMemoryAfter = getFreeDeviceMemory(); | ||
| handleOfNewThread = *handle; | ||
| TLLM_CUDA_CHECK(cublasGetStream(*handle, &streamOfNewThread)); | ||
| }); | ||
|
|
||
| ASSERT_EQ(handleOfNewThread, handleOfExitedThread); | ||
| EXPECT_EQ(streamOfNewThread, nullptr); | ||
| EXPECT_EQ(freeMemoryAfter, freeMemoryBefore); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] |
||
| } | ||
|
|
||
| TEST(CublasHandlePoolTest, LiveThreadsGetDistinctCublasHandlesAndEachThreadKeepsItsOwn) | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Nice to have] Both tests are sequential with at most two live handles, so |
||
| { | ||
| // Leave one idle handle: the main thread takes it, so the other thread must get a different one. | ||
| runInNewThread([] { tensorrt_llm::getCublasHandle(); }); | ||
| auto const handle = tensorrt_llm::getCublasHandle(); | ||
| EXPECT_EQ(*tensorrt_llm::getCublasHandle(), *handle); | ||
|
|
||
| cublasHandle_t handleOfOtherThread{}; | ||
| runInNewThread([&] { handleOfOtherThread = *tensorrt_llm::getCublasHandle(); }); | ||
|
|
||
| EXPECT_NE(handleOfOtherThread, *handle); | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[Required] This is a private copy of PyTorch's
DeviceThreadHandlePool, and PyTorch already pools per-thread cuBLAS/cuBLASLt handles for these callers: every caller ofgetCublasHandle()/getCublasLtHandle()is a thop op inth_common, which links torch, and the same thread holds PyTorch's pooled handle for itsF.linearcalls.at::cuda::getCurrentCUDABlasHandle()/getCurrentCUDABlasLtHandle()also leak the pool and take the cuBLAS workspace from the caching allocator, so KV-cache sizing sees it; the repo already moved DeepGEMM to the PyTorch-owned cuBLASLt handle (DG_USE_PYTORCH_CUBLASLT_HANDLEincpp/tensorrt_llm/deep_gemm/CMakeLists.txt). With two pools each thread keeps two cuBLAS handles per GPU, each with a driver-allocated default workspace, and the pools differ in key (device vs CUcontext) and reset rule (PyTorch re-applies stream/workspace/math mode on every get), so later fixes have to be made twice. Please either pass PyTorch's handles toCublasMMWrapperfrom the thop callers (non-owning; its FP32cublasGemmEx/stridedBatchedGemmpaths would then set the math mode explicitly, since PyTorch appliesallow_tf32), or say in the description why a separate pool stays and track the switch as a follow-up.