Skip to content

[https://nvbugs/7028062][fix] Fix MiniMax test subprocess imports - #21671

Merged
peihu-nv merged 4 commits into
NVIDIA:mainfrom
peihu-nv:fix-m3-test-rank-import-env-scope
Nov 7, 2026
Merged

peihu-nv merged 4 commits into
NVIDIA:mainfrom
peihu-nv:fix-m3-test-rank-import-env-scope

Conversation

@peihu-nv

@peihu-nv peihu-nv commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Dev Engineer Review

The five MiniMax _rank_job.py variants now align rank-subprocess package lookup with the package resolved by pytest. They add the repository root to PYTHONPATH for a missing or checkout-resolved package, and remove entries pointing to the repository root when pytest resolves an external package. The variants retain similar behavior, with small differences in added-line counts.

The waives.txt changes remove six MiniMax SM103 TP4/TP8 communication-test waivers and a MiniMax environment-scope waiver. This enables those entries to run when selected; it does not establish that they passed. A broader modeling-v2 environment-scope waiver remains. No changed public API is reported. Current-head CI and test results are not available.

QA Engineer Review

tests/integration/test_lists/waives.txt was modified. It removes six communication-test skip entries and the test_accuracy_engines_are_inside_environment_scopes[test_minimax_m3.py] skip entry. The GB300 multi-GPU test list still includes the TP4 and TP8 communication directories. The source changes update subprocess package selection but add no test functions or test IDs. Coverage verdict: needs follow-up, because the configured test selection is present but execution results are not available.

Per-File QA Perspective

  • tests/integration/test_lists/waives.txt — Removes the six MiniMax communication-test skips and one environment-scope skip. The communication tests remain selected by tests/integration/test_lists/test-db/l0_gb300_multi_gpus.yml; no test execution result is established.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/comm/_rank_job.py — Rank subprocesses now follow pytest’s checkout-versus-installed package resolution. Verify both PYTHONPATH branches and the missing-package error path.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/comm/_rank_job.py — Applies the same package-selection behavior for TP8 rank subprocesses. Verify the repository path is removed only when pytest resolves an external package.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/comm/_rank_job.py — Applies the package-selection behavior to the SM107 TP2 subprocess. Verify source-checkout and external-install scenarios.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/comm/_rank_job.py — Applies the package-selection behavior to the SM107 TP4 subprocess. Verify source-checkout and external-install scenarios.
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/comm/_rank_job.py — Applies the package-selection behavior to the SM107 TP8 subprocess. Verify source-checkout and external-install scenarios.

Description

MiniMax collective test ranks can import an unbuilt checkout instead of the installed wheel, causing imports to fail before the tests run. Preserve the parent interpreter’s package selection for the rank subprocesses.

Remove the MiniMax-specific environment-scope waiver, whose underlying fix is already on main. The existing file-level waiver remains, so the scope-test file is still skipped by CI.

Test Coverage

  • MiniMax environment-scope check: passed locally (the file-level waiver remains in CI).
  • Launcher path checks across five target copies: installed wheel, checkout and absent-package scenarios passed.
  • Changed-file pre-commit checks passed; CodeRabbit follow-up review completed with warnings.
  • GB300 collective tests and CI are pending.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
@peihu-nv

peihu-nv commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82594 [ run ] triggered by Bot. Commit: d44e440 Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
@peihu-nv

peihu-nv commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82603 [ run ] triggered by Bot. Commit: ee89de2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82594 [ run ] completed with state ABORTED. Commit: d44e440

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82603 [ run ] completed with state SUCCESS. Commit: ee89de2
/LLM/main/L0_MergeRequest_PR pipeline #68191 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@peihu-nv

peihu-nv commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82661 [ run ] triggered by Bot. Commit: ee89de2 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82661 [ run ] completed with state SUCCESS. Commit: ee89de2
/LLM/main/L0_MergeRequest_PR pipeline #68249 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
@peihu-nv

peihu-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82978 [ run ] triggered by Bot. Commit: 64ea3db Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>
@peihu-nv

peihu-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83056 [ run ] triggered by Bot. Commit: db02509 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #82978 [ run ] completed with state ABORTED. Commit: 64ea3db
/LLM/main/L0_MergeRequest_PR pipeline #68535 completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Either:

  • Wait for the PR to be fully approved — the label is added automatically once approval is complete. Having unresolved open comments is fine, or
  • If needed, ask a member of NVIDIA/trt-llm-ci-approvers to add the label manually.
    Then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83056 [ run ] completed with state FAILURE. Commit: db02509
/LLM/main/L0_MergeRequest_PR pipeline #68611 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@peihu-nv peihu-nv changed the title [https://nvbugs/7028062][fix] Restore modeling v2 test coverage [https://nvbugs/7028062][fix] Fix MiniMax test subprocess imports Nov 6, 2026
@peihu-nv
peihu-nv marked this pull request as ready for review November 6, 2026 17:32
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Rank subprocesses now match their PYTHONPATH to pytest’s resolved tensorrt_llm package. The skip list no longer contains entries for six SM103 communication tests or one accuracy-engine environment-scope test.

Changes

MiniMax M3 test execution

Layer / File(s) Summary
Rank subprocess import-path selection
tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/comm/_rank_job.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/comm/_rank_job.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/comm/_rank_job.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/comm/_rank_job.py, tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/comm/_rank_job.py
Each rank job checks where pytest resolves tensorrt_llm. If it is unavailable or resolves inside the repository, the job verifies the repository package directory and prepends the repository root to PYTHONPATH. If it resolves outside the repository, the job removes repository-root entries from PYTHONPATH.
MiniMax M3 skip-list entries
tests/integration/test_lists/waives.txt
The six SM103 TP4/TP8 communication-test entries and the accuracy-engine environment-scope entry were removed. The broader modeling-v2 environment-scope skip remains.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · up to db025

No concrete change-related failure remains established. The change is mergeable after normal checks; investigate the reported CI failure if it persists.

Pre-merge checks: Passed 5

Check name Status Explanation
Title check Passed The title clearly identifies the MiniMax subprocess import fix.
Description check Passed The description explains the issue, solution, waiver changes, and test coverage. It includes the required checklist.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 5 functions across 5 files.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 406fa0a and db02509.

Files selected for processing: 6

  • tests/integration/test_lists/waives.txt
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp4/comm/_rank_job.py
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_103__tp8/comm/_rank_job.py
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp2/comm/_rank_job.py
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp4/comm/_rank_job.py
  • tests/unittest/_torch/modeling_v2/models/minimax_m3/minimax_m3_nvfp4__sm_107__tp8/comm/_rank_job.py
Files with no reviewable changes: 1

  • tests/integration/test_lists/waives.txt

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@sunnyqgg sunnyqgg left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of db02509: 0 blockers, 1 possibly blocking, 0 required, 4 nice to have. The import fix is correct for CI's layout (an unbuilt checkout at /tmp/TensorRT-LLM/src next to an installed wheel). It uses the same check #20814 put into the shared modeling_v2/comm/_rank_job.py, so pytest resolves the wheel and the ranks stop importing the source tree: that tree is where the No module named 'tensorrt_llm.bindings' failures in post-merge 3048/3049/3051 came from. All five stale copies are fixed. Dropping the nvbug 7027434 env-scope waive is also fine. #20623 is in the base and the scope check passes for test_minimax_m3.py. The file-level 7024242 waive still has to stay, because test_glm_moe_dsa.py:117 fails the same check. The only failure in L0 #68611 is the unrelated TestNanoV3Omni::test_auto_dtype[fp8_mmmu_encoder_cuda_graph] accuracy case, which passed in #68535. The open item is a GB300 run of the un-waived cases (see the comment on waives.txt).

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Automatically added "ci: full pre-merge approved" because this PR has satisfied the required GitHub review approvals. Unresolved review conversations and other required checks remain independent merge requirements.

@peihu-nv

peihu-nv commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast --extra-stage "GB300-4_GPUs-PyTorch-Post-Merge-1,GB300-4_GPUs-PyTorch-Post-Merge-2,GR100-PyTorch-Post-Merge-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83143 [ run ] triggered by Bot. Commit: db02509 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #83143 [ run ] completed with state SUCCESS. Commit: db02509
/LLM/main/L0_MergeRequest_PR pipeline #68696 completed with status: 'SUCCESS'

CI Report

Link to invocation

@peihu-nv
peihu-nv merged commit 9529c83 into NVIDIA:main Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants