Skip to content

[JAX] Add EP borrowed-comm path via XLA FFI collectives extension - #3452

Merged
phu0ngng merged 4 commits into
NVIDIA:mainfrom
phu0ngng:ep_jax_single_process
Sep 15, 2026
Merged

phu0ngng merged 4 commits into
NVIDIA:mainfrom
phu0ngng:ep_jax_single_process

Conversation

@phu0ngng

@phu0ngng phu0ngng commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Adds an EP bootstrap path that borrows JAX/XLA's own NCCL communicator instead of self-hosting one via ncclCommInitRank, avoiding a second communicator per EP domain. Auto-selected when the XLA-borrowed NCCL comm is available (build + installed JAX/jaxlib support it); otherwise falls back to self-hosted NCCL.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • ffi_collectives.h/.cpp: generic helper to request/fetch XLA's communicator from any FFI handler, gated behind XLA_FFI_COLLECTIVES_AVAILABLE (__has_include) since not all jaxlib builds ship the required header.
  • ep.cpp/ep.py: a one-shot bootstrap op fetches the borrowed comm and initializes EPBackend; per-step dispatch/combine ops are unchanged.
  • tex.ep.use_nccl_comm_from_xla() auto-selects the path; switching paths mid-process is rejected with an error.
  • No atexit teardown on the borrowed-comm path (XLA may already be tearing down its comm at exit) - use ep_finalize() for mid-process teardown there.

Testing

  • TestEPBorrowedComm reruns the TestEP suite on the borrowed-comm path (smoke subset by default, full suite at L2).
  • TestEpCommSelection: GPU-free unit tests for path-selection gating.
  • qa/L2_jax_distributed_unittest/test.sh runs the borrowed-comm suite via the multi-process EP launcher.

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@phu0ngng
phu0ngng force-pushed the ep_jax_single_process branch from 29d4ed6 to e081e4c Compare September 9, 2026 17:44
@phu0ngng
phu0ngng marked this pull request as ready for review September 9, 2026 17:46
@greptile-apps

greptile-apps Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; no actionable new defects remain, and the previously reported exit-time borrowed-backend shutdown issue is resolved.

Summary

Adds a JAX expert-parallel bootstrap path that borrows an XLA-owned NCCL communicator when the required build headers and JAX runtime support are available.

  • Adds generic XLA FFI helpers for requesting and fetching communicators.
  • Initializes and finalizes the EP backend without taking ownership of borrowed NCCL communicators.
  • Retains the existing self-hosted communicator as an automatic compatibility fallback.
  • Adds communicator-selection tests and a separate multiprocess borrowed-communicator suite.
  • Replaces the self-hosted exit callback with an exit-safe variant that does not shut down a subsequently borrowed backend.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ep_bootstrap] --> B{Borrowed path built and runtime supported?}
    B -->|No| C[Exchange NCCL unique ID]
    C --> D[ncclCommInitRank]
    D --> E[Initialize EP backend]
    B -->|Yes| F[Build EP replica groups]
    F --> G[XLA FFI prepare requests clique]
    G --> H[XLA FFI execute fetches communicator]
    H --> I[Initialize EP backend using borrowed communicator]
    E --> J[Existing EP dispatch and combine operations]
    I --> J
    J --> K[ep_finalize]
    K --> L[Clear JAX executable caches]
    L --> M[Shut down EP backend]
    M --> N{Communicator ownership}
    N -->|Self-hosted| O[Destroy owned NCCL communicator]
    N -->|Borrowed| P[Leave communicator lifecycle to XLA]
Loading

Reviews (4) · Last reviewed commit: "Merge branch 'main' into ep_jax_single_p..."

@phu0ngng
phu0ngng requested a review from tdophung September 9, 2026 20:31
@phu0ngng

phu0ngng commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 JAX

Comment thread tests/jax/test_multi_process_ep.py Resolved
Comment thread tests/jax/test_multi_process_ep.py Resolved

@jberchtold-nvidia jberchtold-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a review pass and everything LGTM besides existing comments from Teddy. I appreciate the thorough version handling in this between pre- and post- this feature landing in XLA. Thanks!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@phu0ngng phu0ngng Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Michael, good suggestion.
XLA has an equivalent (xla::ffi::Collectives/Communicator in collectives_ffi.h), but there is a blocker: jaxlib doesn't package that header yet. Besides, XLA's version has no Support() override, so it requires an exact major.minor match. Left a TODO in the file - I will send the fixes to jaxlib and XLA, then will drop our copy.
Deferring for now since this is not a blocker.

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
…truct init

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
phu0ngng force-pushed the ep_jax_single_process branch from e081e4c to 29d4ed6 Compare September 14, 2026 16:39
@phu0ngng

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 JAX

Signed-off-by: Phuong Nguyen <phuonguyen@nvidia.com>
@phu0ngng
phu0ngng force-pushed the ep_jax_single_process branch from 29d4ed6 to 7f5ebc3 Compare September 14, 2026 18:55
@phu0ngng

Copy link
Copy Markdown
Collaborator Author

/te-ci L1 JAX

@phu0ngng
phu0ngng requested a review from tdophung September 14, 2026 23:29

@tdophung tdophung left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@phu0ngng
phu0ngng merged commit 73b5970 into NVIDIA:main Sep 15, 2026
11 of 12 checks passed
@phu0ngng
phu0ngng deleted the ep_jax_single_process branch September 15, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants