Repository navigation
Conversation
test_operator is CPU-bound, so running it as one process leaves most cores idle. Add an NVTE_TEST_NUM_SHARDS CMake option that registers N ctest tests, each running one GoogleTest shard of test_operator. Without it, test_operator is still a single test. The L0/L1 C++ unit test scripts use up to 15 shards (an odd count, for load balance) when every GPU has at least 80 GB, since all shards share one GPU. On 4x GB200 this cuts test_operator from 1495 s to about 155 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
kainzhong
marked this pull request as draft
November 19, 2026 00:23
Collaborator
Author
|
/te-ci |
kainzhong
marked this pull request as ready for review
November 19, 2026 18:59
Contributor
[Medium impact] The PR appears safe to merge. SummaryThe PR splits C++ operator tests across several processes to use more CPU cores while limiting GPU memory use.
Reviews (2) · Last reviewed commit: "Shard C++ operator tests across CPU core..." · Reviewed by Greptile |
kainzhong
force-pushed
the
shard_cpp_operator_tests
branch
from
November 19, 2026 19:14
e591fdc to
1bf5d2a
Compare
Collaborator
Author
|
/te-ci |
Member
|
Looks like a very nice reduction in time. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CPU is the bottleneck for C++ operator tests, so we should parallelize them to multiple CPUs. Previously we did run them in parallel but we do this by starting a new process for each test which turns to be slow. Later I made all tests run within a single process which is faster, but we now aren't fully utilizing all available hardware resources. This PR now partitions all C++ operator tests to min(15, cores) shards and each shards only launches one process, so we can parallelize with one CPU per process.
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: