Skip to content

fix(cross_entropy): keep the denominator finite when a block is entirely -inf - #3891

Open
lxcxjxhx wants to merge 3 commits into
NVIDIA:mainfrom
lxcxjxhx:fix/cross-entropy-all-neg-inf-block
Open

lxcxjxhx wants to merge 3 commits into
NVIDIA:mainfrom
lxcxjxhx:fix/cross-entropy-all-neg-inf-block

Conversation

@lxcxjxhx

@lxcxjxhx lxcxjxhx commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Description

Restructure only — no code change. The body now follows this repository's template so
the checklist can be answered rather than implied; the technical content is unchanged
from what was already reviewed and approved.

Fixes the -inf handling of the online-softmax merge in the cross-entropy kernels: a
shard that is entirely masked makes global_m == -inf, and exp(m - global_m) then
computes exp(-inf - -inf), which is nan, poisoning the loss and every gradient
through it.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • add _softmax_shift(maximum): returns 0.0 when the running maximum is -inf, the
    maximum otherwise
  • use it in cross_entropy_forward_kernel, cross_entropy_tp_pre_kernel and
    cross_entropy_tp_post_kernel in place of the raw maximum, so a fully masked block
    contributes zero mass instead of nan
  • tests/pytorch/test_cross_entropy.py::test_parallel_cross_entropy_leading_neg_inf_block:
    a 32k-vocabulary row whose first 32 760 logits are -inf
  • tests/pytorch/distributed/test_parallel_cross_entropy.py::test_parallel_cross_entropy_tensor_parallel_leading_neg_inf:
    three ranks where two hold entirely masked shards, compared against single-rank
    torch.nn.functional.cross_entropy

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas — the helper carries a one-line comment saying why shifting by zero preserves both -inf mass and NaN
  • I have made corresponding changes to the documentation — not applicable, the API is unchanged
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes — te-ci on this commit: L0_pytorch_unittest and L0_pytorch_debug_unittest green on A100, H100 and B200, plus lint and the base/devel images. Separately I re-derived the merge arithmetic in torch on CPU and compared both forms against logsumexp: the old form gives nan for the leading -inf case, the new one gives the reference value, and a NaN inside the masked region still propagates.

Note on the status of this PR

blossom-ci has been pending ("CI dispatch accepted") since 2026-11-21T00:29, the
same minute every te-ci context reported. The commit's combined status is pending
because of that one context; nothing in the test results is red.

@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Nov 18, 2026
@greptile-apps

greptile-apps Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues remain.

Summary

The PR keeps the cross-entropy denominator finite while leading blocks or vocabulary shards contain only -inf.

  • Cross-entropy keeps all--inf prefixes from turning later losses into NaN.

No new actionable issues were found. Review was static; GPU tests were not run.

Reviews (5) · Last reviewed commit: "Preserve NaNs when accumulating masked c..." · Reviewed by Greptile

@lxcxjxhx
lxcxjxhx force-pushed the fix/cross-entropy-all-neg-inf-block branch from 89a50ed to 4948e99 Compare November 18, 2026 01:14
@lxcxjxhx

lxcxjxhx commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks — finding 1 was real and is now fixed in the same PR.

Cross-rank merge: cross_entropy_tp_post_kernel had the same shape of bug one level up. When every rank is entirely -inf for a row, global_m is -inf and d * exp(m - global_m) computes exp(-inf - -inf):

         global_m = tl.maximum(m, m_new)
-        d = d * tl.exp(m - global_m) + d_new * tl.exp(m_new - global_m)
+        if global_m == float("-inf"):
+            d = 0.0
+        else:
+            d = d * tl.exp(m - global_m) + d_new * tl.exp(m_new - global_m)

Finding 2 (missing test): added test_parallel_cross_entropy_leading_neg_inf_block in tests/pytorch/test_cross_entropy.py — two rows, 32776 logits with the first 32768 at -inf and the last eight at zero, comparing both the unreduced loss and the input gradient against torch.nn.functional.cross_entropy. It covers the single-rank path only; the merge guard is above that, because reaching it needs a multi-rank launch, so it is read-and-argued rather than tested here.

Comment thread tests/pytorch/test_cross_entropy.py Resolved
Comment thread tests/pytorch/test_cross_entropy.py Resolved
…ely -inf

Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
@lxcxjxhx
lxcxjxhx force-pushed the fix/cross-entropy-all-neg-inf-block branch from f5bc4ad to 4f70e39 Compare November 18, 2026 01:31
@lxcxjxhx

Copy link
Copy Markdown
Contributor Author

Added the tensor-parallel coverage that was missing, in tests/pytorch/distributed/test_parallel_cross_entropy.py:

test_parallel_cross_entropy_tensor_parallel_leading_neg_inf spawns two ranks over a vocabulary of 32 where rank 0's entire shard is -inf and rank 1's is zero, then compares the merged loss and the per-rank gradients against a single-rank torch.nn.functional.cross_entropy over the full vocabulary. That is the case where the local pre-kernel's d would be NaN without the block guard, and it is the closest a test can get to the merge without an all--inf row (see below).

One thing about the merge guard worth stating plainly, because it bounds what a test could claim: when every rank is -inf for a row, the row's softmax is undefined and the loss is NaN in any implementation, including PyTorch — so the merge guard is not there to produce a meaningful loss for that row, it is there so the intermediate d does not become NaN through 0 * exp(-inf - -inf) and can be reasoned about. The test above deliberately uses a row with mass on one rank only, which is the case that has a well-defined answer.

@ptrendx

ptrendx commented Nov 20, 2026

Copy link
Copy Markdown
Member

/te-ci pytorch

@ptrendx ptrendx self-assigned this Nov 20, 2026
@lxcxjxhx

Copy link
Copy Markdown
Contributor Author

Status check on the /te-ci pytorch run you dispatched, in case it is the last thing holding this PR:

Every te-ci context came back green on 2026-11-21T00:28–00:29, on all three GPU generations:

te-ci/L0_pytorch_unittest--A100_1GPU          success
te-ci/L0_pytorch_unittest--H100_1GPU          success
te-ci/L0_pytorch_unittest--B200_1GPU          success
te-ci/L0_pytorch_debug_unittest--H100_1GPU    success
te-ci/L0_pytorch_debug_unittest--B200_1GPU    success
te-ci/L0_pytorch_lint                         success
te-ci/pytorch-py3-base-amd64                  success
te-ci/pytorch-py3-devel-amd64                 success
te-ci                                         success   (84111794 Complete)

The only context that is not finished is blossom-ci, which has been sitting at pending — "CI dispatch accepted !!!" — since 2026-11-21T00:29, i.e. since the same minute the rest of the run reported. That single stuck status is what keeps the commit's combined state at pending and the PR at blocked; nothing in the test results is red.

If that dispatch is genuinely still running, ignore this. If it is a dispatch that never reported back, a re-dispatch (or dismissing it) should be all that is left — the tensor-parallel coverage you asked about is in the branch and included in those A100/H100/B200 runs.

@lxcxjxhx

Copy link
Copy Markdown
Contributor Author

Since this PR is waiting on a status rather than on review, I re-derived the merge arithmetic on CPU to check that _softmax_shift does what the description claims. It walks the same recurrence (m_new = max(m, block_max), d = d*exp(m - shift) + sum(exp(x - shift))) in float32 in torch, and compares both the old and new forms against logsumexp:

case reference old exp(m - m_new) new exp(m - shift)
ordinary row 3.799424 3.799424 3.799424
leading -inf blocks, 64 finite 1.654771 nan 1.654771
entire row -inf nan nan nan
masked prefix containing a NaN nan nan nan (preserved)

and the two-rank merge of cross_entropy_tp_post, with rank 1 entirely masked:

merged m merged d loss
old -inf nan nan
new -inf 0.0 -inf
reference (single rank) — — 2.138920 (old/new both 2.138921 when rank 0 is finite)

So the two properties the change is for both hold in the replica: a fully masked shard contributes zero mass instead of poisoning the reduction with nan, and a nan that is inside the masked region still propagates (last row of the table) rather than being erased by the same where.

Caveat on what this is worth: it is a replica of the recurrence in torch, not the Triton kernel — it says the arithmetic is right, not that the compiled kernel is. The te-ci run already passed on A100/H100/B200, so between the two the arithmetic and the codegen are each covered by something. Probe kept at _te_ce_merge_probe.py (torch, CPU, ~1 s) if anyone wants to extend the case list.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants