Skip to content

Shard C++ tests to multiple CPUs - #3899

Open
kainzhong wants to merge 1 commit into
NVIDIA:mainfrom
kainzhong:shard_cpp_operator_tests
Open

kainzhong wants to merge 1 commit into
NVIDIA:mainfrom
kainzhong:shard_cpp_operator_tests

Conversation

@kainzhong

@kainzhong kainzhong commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

CPU is the bottleneck for C++ operator tests, so we should parallelize them to multiple CPUs. Previously we did run them in parallel but we do this by starting a new process for each test which turns to be slow. Later I made all tests run within a single process which is faster, but we now aren't fully utilizing all available hardware resources. This PR now partitions all C++ operator tests to min(15, cores) shards and each shards only launches one process, so we can parallelize with one CPU per process.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Shard C++ tests on CPU (only do it with >=80GB GPU memory and will keep shard number an odd number)

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

test_operator is CPU-bound, so running it as one process leaves most cores idle.
Add an NVTE_TEST_NUM_SHARDS CMake option that registers N ctest tests, each
running one GoogleTest shard of test_operator. Without it, test_operator is
still a single test.

The L0/L1 C++ unit test scripts use up to 15 shards (an odd count, for load
balance) when every GPU has at least 80 GB, since all shards share one GPU.
On 4x GB200 this cuts test_operator from 1495 s to about 155 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
@kainzhong
kainzhong marked this pull request as draft November 19, 2026 00:23
@kainzhong

Copy link
Copy Markdown
Collaborator Author

/te-ci

@kainzhong
kainzhong marked this pull request as ready for review November 19, 2026 18:59
@greptile-apps

greptile-apps Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

The PR splits C++ operator tests across several processes to use more CPU cores while limiting GPU memory use.

  • C++ operator tests split across processes on large-memory GPU runners.

Reviews (2) · Last reviewed commit: "Shard C++ operator tests across CPU core..." · Reviewed by Greptile

Comment thread qa/L0_pytorch_unittest/test.sh Resolved
Comment thread qa/L0_jax_unittest/test.sh Resolved
Comment thread qa/L0_jax_unittest/test.sh Resolved
Comment thread qa/gpu_test_queue.sh Resolved
@kainzhong kainzhong changed the title Shard C++ tests to multiple CPUs & python tests to all GPUs Shard C++ tests to multiple CPUs Nov 19, 2026
@kainzhong
kainzhong force-pushed the shard_cpp_operator_tests branch from e591fdc to 1bf5d2a Compare November 19, 2026 19:14
@kainzhong

Copy link
Copy Markdown
Collaborator Author

/te-ci

@ptrendx

ptrendx commented Nov 21, 2026

Copy link
Copy Markdown
Member

Looks like a very nice reduction in time.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants