Repository navigation
Conversation
[Medium impact] The PR appears safe to merge; no blocking issues remain. SummaryThe PR keeps the cross-entropy denominator finite while leading blocks or vocabulary shards contain only
No new actionable issues were found. Review was static; GPU tests were not run. Reviews (5) · Last reviewed commit: "Preserve NaNs when accumulating masked c..." · Reviewed by Greptile |
89a50ed to
4948e99
Compare
|
Thanks — finding 1 was real and is now fixed in the same PR. Cross-rank merge: global_m = tl.maximum(m, m_new)
- d = d * tl.exp(m - global_m) + d_new * tl.exp(m_new - global_m)
+ if global_m == float("-inf"):
+ d = 0.0
+ else:
+ d = d * tl.exp(m - global_m) + d_new * tl.exp(m_new - global_m)Finding 2 (missing test): added |
…ely -inf Signed-off-by: lxcxjxhx <lxcxjxhx@users.noreply.github.com>
f5bc4ad to
4f70e39
Compare
|
Added the tensor-parallel coverage that was missing, in
One thing about the merge guard worth stating plainly, because it bounds what a test could claim: when every rank is |
for more information, see https://pre-commit.ci
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
|
/te-ci pytorch |
|
Status check on the Every The only context that is not finished is If that dispatch is genuinely still running, ignore this. If it is a dispatch that never reported back, a re-dispatch (or dismissing it) should be all that is left — the tensor-parallel coverage you asked about is in the branch and included in those A100/H100/B200 runs. |
|
Since this PR is waiting on a status rather than on review, I re-derived the merge arithmetic on CPU to check that
and the two-rank merge of
So the two properties the change is for both hold in the replica: a fully masked shard contributes zero mass instead of poisoning the reduction with Caveat on what this is worth: it is a replica of the recurrence in torch, not the Triton kernel — it says the arithmetic is right, not that the compiled kernel is. The |
Description
Restructure only — no code change. The body now follows this repository's template so
the checklist can be answered rather than implied; the technical content is unchanged
from what was already reviewed and approved.
Fixes the
-infhandling of the online-softmax merge in the cross-entropy kernels: ashard that is entirely masked makes
global_m == -inf, andexp(m - global_m)thencomputes
exp(-inf - -inf), which isnan, poisoning the loss and every gradientthrough it.
Type of change
Changes
_softmax_shift(maximum): returns0.0when the running maximum is-inf, themaximum otherwise
cross_entropy_forward_kernel,cross_entropy_tp_pre_kernelandcross_entropy_tp_post_kernelin place of the raw maximum, so a fully masked blockcontributes zero mass instead of
nantests/pytorch/test_cross_entropy.py::test_parallel_cross_entropy_leading_neg_inf_block:a 32k-vocabulary row whose first 32 760 logits are
-inftests/pytorch/distributed/test_parallel_cross_entropy.py::test_parallel_cross_entropy_tensor_parallel_leading_neg_inf:three ranks where two hold entirely masked shards, compared against single-rank
torch.nn.functional.cross_entropyChecklist:
-infmass andNaNte-cion this commit:L0_pytorch_unittestandL0_pytorch_debug_unittestgreen on A100, H100 and B200, plus lint and the base/devel images. Separately I re-derived the merge arithmetic in torch on CPU and compared both forms againstlogsumexp: the old form givesnanfor the leading-infcase, the new one gives the reference value, and aNaNinside the masked region still propagates.Note on the status of this PR
blossom-cihas beenpending("CI dispatch accepted") since 2026-11-21T00:29, thesame minute every
te-cicontext reported. The commit's combined status ispendingbecause of that one context; nothing in the test results is red.