Repository navigation
Conversation
|
/bot run |
|
PR_Github #85455 [ run ] triggered by Bot. Commit: |
|
PR_Github #85455 [ run ] completed with state
|
|
/bot run |
|
PR_Github #85801 [ run ] triggered by Bot. Commit: |
|
PR_Github #85801 [ run ] completed with state |
|
No actionable comments were generated in the recent review. WalkthroughThe change adds a shared method to calculate unfused attention context workspace sizes. The attention layout uses the method, and unit tests cover self-attention, cross-attention, and grouped-query sizing. ChangesAttention Workspace Sizing
Estimated code review effort: 3 (Moderate) | ~20 minutes Change: Refactor Suggested reviewers: Merge Risk:
|
|
@pengbowang-nv, could you help to review this PR as trt-llm-torch-attention-devs? Thanks~ Assignment explanation
Affinity and workload are snapshots at selection, not live counters. Existing reservations on this PR are counted once. |
mikeiovine
left a comment
There was a problem hiding this comment.
Stamp on behalf of runtime devs, delegating proper review to @NVIDIA/trt-llm-torch-attention-devs
There was a problem hiding this comment.
Thanks for the PR. It moves the unfused context-attention workspace sizing out of MHA::contextWorkspaceLayout into a new host-only component, tllm::common_attention_workspace, so the workspace gtest no longer needs the product library.
Reviewed at commit 3c7fad70d0 (9 files, +211/−41).
The overlap with #21915 was checked against that PR's head c64c037.
Approve: no Blocker found. 6 suggestions below are nice to have.
Nice to have (6): 6 inline, 0 below
- (N1: Architecture and design) AttentionUnfusedContextWorkspaceParams takes numHeads and numAttnHeads, which the only caller always passes equal; different values would size the Q and score buffers from different head counts unchecked (
attentionWorkspace.h:83) - (N1: Architecture and design) A new static library, facade entry and PUBLIC common_src link for one 30-line function whose siblings are inline in attentionWorkspace.h; no common_src source uses it (
CMakeLists.txt:95) - (N3: Missing or ineffective tests) The ragged self/cross tests copy the helper's formulas and compare 256-byte-aligned totals, so packed-count regressions in the mask, score and padding sizes still pass (
attentionWorkspaceTest.cpp:65) - (R1: PR process) Open PR #21915 deletes this hunk and re-implements the unfused sizing inline in UnfusedFmha::contextWorkspaceLayout; the two PRs need an agreed merge order (
attentionOp.cpp:806) - (N4: Performance optimization opportunity) Unfused self-attention now reserves an encoderPaddingOffset slice (4 x batch x seq bytes) that the old code sized at 0 and only cross attention reads (
attentionWorkspace.cpp:50) - (N5: Code tidiness) The zero-token early return is unreachable from production and returns a zero cuBLAS slice that the caller's layout does not match; the padded-size rationale comment was dropped (
attentionWorkspace.cpp:28)
What this review checked
- Every unfused slice except encoderPaddingOffset (attention mask, Q/K/V, QK, QKV, fp32 scores, padding offsets, tokens info, cu-seqlens) matches the deleted inline formulas. The fused path keeps its 32 MiB cuBLAS slice and its separate-Q/KV, FP8, Sage and multi-CTA sizes unchanged.
- enqueueContext carves the workspace with the same contextWorkspaceLayout(p) it is sized with (attentionOp.cpp:1020, 1034), and zero-token input never reaches it, so the new early return cannot leave a running context call without a cuBLAS slice.
- CUBLAS_WORKSPACE_SIZE keeps one definition and its value (workspace.h:22), and all 14 uses still reach it through cudaUtils.h. attentionWorkspaceTest runs pre-merge in test_unit_tests[common-90] on H100, and the unfused path runs pre-merge in the T5 end-to-end tests on B200.
- Not covered: No GPU run and no full CMake build; the helper and a mutation harness were compiled on the host with stub CUDA headers.
- Not covered: The T5 and other unfused-path end-to-end tests were not rerun; the PR's blossom-ci L0 run on 3c7fad7 passed.
Automated review by trtllm-agent-review. If a point is wrong, reply in its thread. After you push fixes or reply, the bot re-checks the Blockers within a few hours.
| size_t querySequenceLength{}; | ||
| size_t kvSequenceLength{}; | ||
| size_t packedTokenCount{}; | ||
| int numHeads{}; |
There was a problem hiding this comment.
[Nice to have] (N1: Architecture and design) numHeads sizes qkBuf/qkFloatBuf (attentionWorkspace.cpp:38) and numAttnHeads sizes qBuf/qkvBuf (attentionWorkspace.cpp:36), but nothing says how they differ. The only caller passes .numHeads = static_cast<int>(num_heads) and .numAttnHeads = mNumAttnHeads (attentionOp.cpp:811-812), and TrtllmAttention sets mNumAttnHeads = static_cast<int>(num_heads) because neither TP nor CP applies to this op (attentionVariant.h:54-55). Every new test passes equal values too. A caller that passes different values gets the Q and score buffers sized from different head counts, with no check. Could this be one numQueryHeads next to numAttnKvHeads? If two counts are really needed, please document each field with //!< and say which buffers use which.
There was a problem hiding this comment.
refactor: use one query head count for attention workspace sizing
| target_compile_definitions(tllm_common_cublas PUBLIC ENABLE_CUBLASLT_FP4_GEMM) | ||
| endif() | ||
|
|
||
| add_library(tllm_common_attention_workspace STATIC attentionWorkspace.cpp) |
There was a problem hiding this comment.
[Nice to have] (N1: Architecture and design) buildUnfusedContextSizes is the only out-of-line member of AttentionWorkspaceManager: buildContextLayout, materializeContext and the other builders are inline in attentionWorkspace.h (244-395). Here it gets a new STATIC archive, a TLLM_FACADE_COMPONENT_TARGETS entry and a PUBLIC link from common_src (lines 126-128). None of common_src's sources include attentionWorkspace.h, and the facade entry already whole-archives the library into libtensorrt_llm. Because ${SHARED_TARGET} links common_src PUBLIC, the archive also lands on the link line of every libtensorrt_llm consumer. It also links tllm::common_error only to get its include directories; the helper uses none of that library's symbols. Either define the function inline next to its siblings and link the test to a header-only target, or keep the component and drop the common_src edge, depending on tllm::common_headers plus the CUDA include directories instead.
There was a problem hiding this comment.
Narrowed attention workspace component dependencies.
| constexpr int32_t kPackedTokenCount = 14; | ||
| constexpr int32_t kHeadSize = 8; | ||
|
|
||
| size_t expectedUnfusedContextWorkspace(bool crossAttention) |
There was a problem hiding this comment.
[Nice to have] (N3: Missing or ineffective tests) expectedUnfusedContextWorkspace restates the helper's formulas line by line, and both ragged tests compare only buildContextLayout(...).totalSize, where every slice is rounded up to 256 bytes. At batch 2, length 11 and 14 packed tokens with all head counts 1, a packed-count mistake often lands in the same bucket: attentionMask is 484 B padded and 308 B packed, and both round to 512. I built the helper on the host and tried nine mutations. RaggedUnfusedSelfAttentionUsesPaddedTokenCounts still passes with attentionMask, qkBuf, paddingOffset or encoderPaddingOffset sized by the packed count, and with the scores sized by the KV head count. UnfusedGroupedQuerySizesDistinguishQueryAndKvHeads catches all nine, including every one the two ragged tests catch. Suggest dropping the two ragged tests, or giving them per-field literal expectations at sizes well above 256 B (for example batch 4, length 64, 100 packed tokens) with distinct Q and KV head counts.
There was a problem hiding this comment.
test: check ragged attention workspace buffer sizes directly
| // smaller than batch_size * max(context q length), so sizing by max_num_tokens underestimates. | ||
| size_t const attention_mask_size = mEnableContextFMHA ? 0 : size * batch_size * input_seq_length * kv_seq_length; | ||
| auto workspaceSizes = mEnableContextFMHA ? AttentionContextWorkspaceSizes{} | ||
| : AttentionWorkspaceManager::buildUnfusedContextSizes({.elementSize = size, |
There was a problem hiding this comment.
[Nice to have] (R1: PR process) This hunk overlaps open PR #21915 ("Phased FMHA libraries, selected in Python"). At its head c64c037, MHA::contextWorkspaceSizes (thop/attentionOp.cpp:320) sizes only the fused buffers. A new UnfusedFmha::contextWorkspaceLayout (thop/phasedFmha.cpp:2525-2557) re-implements the attention mask, Q/K/V, score and padding-offset sizes inline, with the old "must upper-bound the carve" comment. If #21915 lands first, this change has to be redone there. If this PR lands first and #21915 keeps its inline copy when it rebases, buildUnfusedContextSizes has no production caller left, and the new host tests check sizing that production no longer uses. Could you link the two PRs and agree the order with the #21915 author, ideally so that UnfusedFmha::contextWorkspaceLayout calls this helper?
| sizes.qkvBuf = sizes.qBuf; | ||
| sizes.qkFloatBuf = sizeof(float) * scoreElements; | ||
| sizes.paddingOffset = sizeof(int) * paddedQueryTokens; | ||
| sizes.encoderPaddingOffset = sizeof(int) * paddedKvTokens; |
There was a problem hiding this comment.
[Nice to have] (N4: Performance optimization opportunity) For self-attention, kvSequenceLength is input_seq_length (attentionOp.cpp:804), so this now reserves 4 * batch * input_seq_length bytes. The code it replaces used sizeof(int) * batch_size * cross_kv_length, and crossKvLength() is 0 unless the op is cross attention (attentionFeature.h:668-675). The slice is only read through cross.active() ? workspaceViews.encoderPaddingOffset : nullptr (attentionOp.cpp:1054) and in the cross-only branch (attentionOp.cpp:1435). So every unfused self-attention context call reserves workspace nothing reads; that covers T5's relative-position attention, fp32, and head sizes without an FMHA kernel. RaggedUnfusedSelfAttentionUsesPaddedTokenCounts expects the larger total, so restoring the old size fails it. Suggest passing the cross KV length in the params and sizing encoderPaddingOffset = sizeof(int) * batchSize * crossKvLength (0 for self-attention), then updating the expectation.
There was a problem hiding this comment.
fix: reserve encoder padding workspace only for cross attention
| AttentionUnfusedContextWorkspaceParams const& params) noexcept | ||
| { | ||
| AttentionContextWorkspaceSizes sizes{}; | ||
| if (params.packedTokenCount == 0) |
There was a problem hiding this comment.
[Nice to have] (N5: Code tidiness) Two small points on the helper:
- This branch has no production trigger.
getWorkspaceSizeForContextreturns 0 fornum_tokens == 0before it builds the layout (attentionOp.cpp:884), and the context call only runs when there are context tokens. The branch also returnscublasWorkspace = 0, while the caller then re-fillscuQSeqlens,cuKvSeqlens,cuMaskRows,rotaryInvFreqandtokensInfo, so its "empty input needs no workspace" contract is not whatMHA::contextWorkspaceLayoutbuilds. Consider dropping it together withEmptyUnfusedContextNeedsNoWorkspace, or assertingpackedTokenCount > 0. - The removed comment explained why these buffers use padded counts: with padding removal, the packed token count can be smaller than
batch_size * max(context q length). The new doc comment (attentionWorkspace.h:240) only says "padded". One sentence there would keep the reason next to the formulas.
There was a problem hiding this comment.
- fix: remove unused empty attention workspace sizing path
- docs: explain padded attention workspace sizing
Extract padded unfused-context workspace sizing into the PIC C++20 static tllm::common_attention_workspace library. Keep geometry and element size in AttentionUnfusedContextWorkspaceParams and retain noexcept sizing. Size Q/K/V, score buffers, and padding metadata from padded query/KV counts, while sizing token metadata from the packed count. Apply the production integration to MHA::contextWorkspaceLayout in thop: the target branch replaced the original common/attentionOp.cpp. Preserve the replacement architecture and its fused, FP8, rotary, and Sage sizing. Keep the empty-input early return and explicitly convert its head count to the sizing interface type. Replace attentionWorkspace.h CUDA helper dependencies with CUDA type headers and move the unchanged CUBLAS_WORKSPACE_SIZE macro into workspace.h. Link the host tests to the sizing component and retain whole-archive product linkage. Preserve the target branch layout/materialization tests and add ragged self/cross-attention, empty-input, and grouped-query cases. Validation: - Build attentionWorkspaceTest in the SM80 RelWithDebInfo devcontainer. - Pass all nine AttentionWorkspaceManagerTest cases through CTest. - Compile thop/attentionOp.cpp with generated project flags and -Werror; confirm its object references buildUnfusedContextSizes. - Verify the test executable has no CUDA runtime or product shared-library dependency. - Pass repository pre-commit checks for all changed files. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Document why Q/K/V and score buffers use padded counts: padding removal can leave fewer packed tokens than batch size times the maximum context query length. Keep the allocation rationale beside the sizing helper declaration. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Leave empty-context handling with MHA::getWorkspaceSizeForContext, which returns zero before building a layout. Remove the sizing helper early return and its dedicated empty-input test, whose zero-workspace contract does not describe the fields subsequently populated by the production layout builder. Validation: rebuilt attentionWorkspaceTest and passed all eight remaining cases with CTest from tests/unit_tests/common. Top-level CTest discovery is blocked by an unrelated ucxCommTest/libpg_utils.so TllmException symbol lookup error. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Pass the cross KV length separately from the general KV sequence length into unfused context sizing. Size encoder padding offsets from the cross extent so self-attention reserves no unused encoder slice while preserving padded K/V buffer sizing and cross-attention offsets. Update the ragged self/cross-attention layout expectations and assert their encoder slice sizes explicitly. Keep grouped-query cross-attention coverage with an explicit cross extent. Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases; compiled the production thop/attentionOp.cpp consumer using its generated project flags. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Replace numHeads and numAttnHeads with numQueryHeads in the unfused context sizing parameters. Use it for both Q/QKV and score buffers, preventing callers from supplying inconsistent query head geometry. Keep the KV head count separate for grouped-query attention. Update the production MHA caller and test initializers. The production query head counts are equal because this attention op has neither tensor nor context parallelism. Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases, including grouped-query buffer assertions; compiled thop/attentionOp.cpp with its generated project flags; passed pre-commit checks. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Replace the duplicated sizing formulas and aligned-total comparisons with literal per-field expectations for ragged self and cross attention. Use batch four, query length 64, 100 packed tokens, eight query heads, and two KV heads to expose packed-count and head-count mistakes. Keep explicit zero encoder padding coverage for self-attention and a distinct cross KV length of 32. Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases. Temporary host mutants confirmed that the ragged tests detect packed attention-mask, score-buffer, and padding-offset sizing, an unwanted self-attention encoder slice, and scores sized with KV heads. Production sources were unchanged during mutation checks. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
Depend directly on common_headers and CUDA toolkit include directories instead of linking common_error solely for its include paths. Remove the unused PUBLIC common_src edge so the workspace archive does not propagate to libtensorrt_llm consumers. Keep the existing private whole-archive facade entry and direct host-test linkage. Validation: rebuilt attentionWorkspaceTest and passed all eight component CTest cases. Verified generated facade linkage retains the workspace archive exactly once with whole-archive, ucxCommTest no longer links that archive, and the host test links neither common_error nor libtensorrt_llm. The workspace archive has no undefined symbols and the host executable has no CUDA runtime dependency. Signed-off-by: Robin Kobus <19427718+Funatiq@users.noreply.github.com>
3c7fad7 to
f418c79
Compare
|
/bot run |
|
PR_Github #86393 [ run ] triggered by Bot. Commit: |
|
PR_Github #86393 [ run ] completed with state
|
|
/bot run |
|
PR_Github #86523 [ run ] triggered by Bot. Commit: |
|
PR_Github #86523 [ run ] completed with state
|
Description
Extract padded unfused-context workspace sizing into the PIC C++20 static tllm::common_attention_workspace library. Keep geometry and element size in AttentionUnfusedContextWorkspaceParams and retain noexcept sizing. Size Q/K/V, score buffers, and padding metadata from padded query/KV counts, while sizing token metadata from the packed count.
Apply the production integration to MHA::contextWorkspaceLayout in thop: the target branch replaced the original common/attentionOp.cpp. Preserve the replacement architecture and its fused, FP8, rotary, and Sage sizing. Keep the empty-input early return and explicitly convert its head count to the sizing interface type.
Replace attentionWorkspace.h CUDA helper dependencies with CUDA type headers and move the unchanged CUBLAS_WORKSPACE_SIZE macro into workspace.h. Link the host tests to the sizing component and retain whole-archive product linkage. Preserve the target branch layout/materialization tests and add ragged self/cross-attention, empty-input, and grouped-query cases.
Dev Engineer Review
AttentionUnfusedContextWorkspaceParamsandAttentionWorkspaceManager::buildUnfusedContextSizes(...) noexceptinattentionWorkspace.h. The sizing logic moves toattentionWorkspace.cpp, built as static librarytllm::common_attention_workspace.tllm::common_headerspublicly. It joinsTLLM_FACADE_COMPONENT_TARGETS. The diff does not changecommon_srclink dependencies.buildUnfusedContextSizeshas no zero-token early return. The diff does not show one. Empty-input handling is not established in this function.MHA::contextWorkspaceLayoutnow calls the new function when context FMHA is off. The inline unfused mask, Q/K/V, QK, QKV, float QK, and padding-offset sizes are removed. Context FMHA uses empty unfused sizes. Separate-Q/KV FMHA setsqBufexplicitly. Fused, FP8, rotary, Sage, and multi-CTA sizing stay in place.CUBLAS_WORKSPACE_SIZE(33554432) moves fromcudaUtils.htoworkspace.h.cudaUtils.hnow includesworkspace.h, so existing includers still see the macro.attentionWorkspace.hdrops thecudaUtils.hinclude and addscuda_fp8.handvector_types.h. Translation units that relied on the removed transitive includes may fail to compile.QA Engineer Review
Test files changed:
cpp/tests/unit_tests/common/attentionWorkspaceTest.cpp(three new tests) andcpp/tests/unit_tests/common/CMakeLists.txt(link target and labels changed tounit;common;attention;host).RaggedUnfusedSelfAttentionUsesPaddedTokenCounts: ragged self-attention with per-field literal expectations.RaggedUnfusedCrossAttentionUsesPaddedTokenCounts: cross-attention, includingencoderPaddingOffsetfrom the cross extent.UnfusedGroupedQuerySizesDistinguishQueryAndKvHeads: grouped-query sizes. Packed count 7 and 15 leave padded buffers andpaddingOffsetunchanged, and onlytokensInfoscales.MHA::contextWorkspaceLayoutintegration, and CI test-db or qa list entries (not established from the diff).Coverage verdict: needs follow-up. The helper is tested directly. The production caller is not.
Per-File QA Perspective
cudaUtils.h: macro removed here. Verify builds still resolve it throughworkspace.h.workspace.h: addsCUBLAS_WORKSPACE_SIZE. Verify the cuBLAS workspace stays at 32 MiB.cpp/tensorrt_llm/CMakeLists.txt: adds the new target to facade components. Verify whole-archive product linking.cpp/tensorrt_llm/common/CMakeLists.txt: adds the static C++20 library. Verify clean builds.attentionWorkspace.cppand.h: new unfused sizing. Verify totals match the previous inline values for ragged, cross-attention, and grouped-query inputs.attentionOp.cpp:contextWorkspaceLayoutuses the shared path. Verify unfused, fused, separate-Q/KV FMHA, FP8, and Sage workspace sizes.cpp/tests/unit_tests/common/CMakeLists.txt: changes link and labels. Verify the test runs underunit;common;attention;host.Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.