Repository navigation
[Common] Add BF16 activation tables and fused dbias to the CuTeDSL register-resident MXFP8 kernel - #3761
[Common] Add BF16 activation tables and fused dbias to the CuTeDSL register-resident MXFP8 kernel#3761tdophung wants to merge 8 commits into
Conversation
|
the gated kernels with LUT and register-resident approach is also added in this PR in CUDA,. However I would like to move it to CuteDSL. However, @kainzhong is already working on it, so this draft will temporarily have the CUDA version, but will rebase don kaining's CuteDSL version of the gated kernel to add the LUT once that PR is up. This will make this PR also waiting for that. If not, I will drop support for LUT in gateed kernel entirely in this PR and make that a follow up PR |
8dae805 to
30849b0
Compare
30849b0 to
39746c1
Compare
39746c1 to
c6759b9
Compare
This is not true anymore as I have moved support for LUT in gated kernel onto #3853 . so this PR should not be blocked by #3840 |
[Medium impact] The PR appears safe to merge; the reported older-GPU test failure is fixed. SummaryAdds opt-in BF16 activation tables and expands the CuTeDSL register-resident MXFP8 kernel.
Reviews (7) · Last reviewed commit: "[Common] Skip the activation table graph..." · Reviewed by Greptile |
892cc55 to
624edea
Compare
kainzhong
left a comment
There was a problem hiding this comment.
Only a few non-critical issues. Btw Claude says NVTE_ACTIVATION_TABLE_EXACT_DSILU isn't tested anywhere because it's neither explicitly covered by qa test scripts nor implicitly covered in your C++ tests (it only covers NVTE_ACTIVATION_TABLE path because you call device_table() directly there)
…gister-resident MXFP8 kernel Add opt-in BF16 lookup tables (NVTE_ACTIVATION_TABLE=1) for GeLU, SiLU and their derivatives in the MXFP8 quantize. TE common builds each table on first use, once per device. The CuTeDSL register-resident kernel reads the tables from shared memory; the CUDA generic kernel and the other CuTeDSL kernels compute the values the tables hold, so both backends write the same bytes. The CuTeDSL register-resident kernel now also takes activation derivatives, on 256-column tiles, with fused dbias and amax, any row count, column tails, and FP16 and FP32 inputs. The CuTeDSL kernels add dbias in the CUDA kernels' row order, except FP32 dbias with rowwise-only output, where CUDA adds row pairs (r, r + 32) first. They give a block of only NaNs the CUDA kernels' scale. test_activation_table checks every BF16 input of each table against the computed activation. test_cast_mxfp8_act_exact checks the fused quantize bit for bit against the activation computed per element and then quantized, with the tables on and off. The C++ QA scripts run test_activation_table, and the dSiLU table cases of test_cast_mxfp8_act_exact, again with NVTE_ACTIVATION_TABLE_EXACT_DSILU=1, which is read once per process. The PyTorch and JAX CuTeDSL backend tests name table-reading kernels as TE does, so they also run with NVTE_ACTIVATION_TABLE=1. Signed-off-by: Teddy Do <tdophung@nvidia.com>
…er-resident MXFP8 kernel The threads that reduce the column amaxes of a tile now also write the colwise scales of their own columns, where the first warp row wrote them after the reduction: - On 256-column tiles, 128 reducers fill four of the eight warps. The other four skip the reduction and the scale stores in a warp-uniform loop instead of running them predicated off. With a column tail, a reducer writes only the columns inside the tensor. - On GEMM-swizzled 512-column tiles outside the tail kernel, every thread reduces, and the scale stores, a byte at a time, are spread over all eight warps instead of issued by warp 0 alone. Compact 512-column scales and the 512-column tail kernel keep the first warp row's write, and FP16 and FP32 inputs keep their own colwise path. Each scale takes the same value and address, so the output bytes do not change. Signed-off-by: Teddy Do <tdophung@nvidia.com>
…d when it reads a table The register-resident kernel launches clusters of CTAs on adjacent column tiles: up to 4 on 512-column tiles of large tensors and up to 8 on 256-column tiles. The kernel uses no cluster-scope feature; co-scheduling adjacent columns raises the rate at which memory delivers a cast's reads. A tile that reads an activation table is bound by its shared-memory lookups instead, so the cluster buys it nothing, while whole clusters can leave CTA slots of the GPU idle. Launch the configs that read a table (BF16 GeLU and SiLU forward) with cluster width 1. Casts and activations without a table keep their cluster widths, and the derivatives already launched unclustered. Only the launch geometry changes, so the output bytes do not change. Signed-off-by: Teddy Do <tdophung@nvidia.com>
CuTeDSL SiLU computes x * sigmoid(x) as util/math.h does, with the IEEE reciprocal of 1 + exp(-x). ptxas guards that reciprocal with a range check and a slow path on every element, which made BF16 SiLU forward without a table slower than computing x / (1 + exp(-x)). sigmoid() gains fast_rcp. For 1 <= y < 2^126, rcp_f32_in_range issues the approximate reciprocal and the one Newton step that rcp.rn.f32 takes in that range, without its range check, so it returns the same bits; 1 + exp(-x) is in that range for every x >= SIGMOID_FAST_RCP_MIN (-80). The register-resident kernel's BF16 SiLU forward without a table checks once per thread row, with min_nan_bf16x2, that every value is at least -80: such rows take the fast reciprocal, and rows holding a smaller value or a NaN keep the division. The output bytes do not change. The other kernels and dSiLU keep the division. test_silu_fast_reciprocal_rows compares SiLU forward without a table with the CUDA backend bit for bit on 512- and 256-column tiles, both directions and colwise only, with both scale layouts. Each 16 columns of a row hold one -100, at another position in each of 16 consecutive rows, so a row check that misses any value of the thread row fails. Signed-off-by: Teddy Do <tdophung@nvidia.com>
624edea to
d31d9a8
Compare
…uTeDSL register-resident MXFP8 kernel Signed-off-by: Teddy Do <tdophung@nvidia.com>
|
/te-ci |
…apability 10.0 Signed-off-by: Teddy Do <tdophung@nvidia.com>
Description
Speeds up the fused MXFP8 activation quantize (GeLU/SiLU, their derivatives, dbias) on the CuTeDSL backend.
The main piece is opt-in BF16 activation tables (
NVTE_ACTIVATION_TABLE=1, off by default). A BF16 input has only 64K values, so GeLU, SiLU and their derivatives can be read from a small table (under 64 KB per device, built on first use) instead of computed. The CuTeDSL register-resident kernel (quantize_mxfp8_g2r_bidim.py) reads the table from shared memory; the CUDA kernel and the other CuTeDSL kernels compute the same values, so both backends still write the same bytes. It's opt-in because the tables differ from the computed activations in two corner cases: GeLU/SiLU of -inf gives -0 instead of NaN, and derivatives of tiny inputs are within 1-2 ulp.NVTE_ACTIVATION_TABLE_EXACT_DSILU=1picks an exact (bigger) dSiLU table.The register-resident kernel also now handles derivatives, fused dbias, amax, column tails, any row count, and FP16/FP32 inputs, so more calls go to it instead of the general kernel.
On top of that, three smaller commits (each measured against the previous one):
rcp.approx+ one Newton step when the whole row is in range, which gives the same bits as the division there, and falls back to the division otherwiseBehavior changes worth knowing:
x * sigmoid(x)likeutil/math.h. The oldx / (1 + exp(-x))differed from CUDA in the last bit for some FP32 inputs. [Common] Add gated MXFP8 CuTeDSL kernel #3840 makes the same change.Perf on GB300 vs main (BF16 in, E4M3 out, both directions, swizzled scales, 4096x18432, median of 500 calls with L2 flushed, same GPU):
dbias + dGeLU (tables on) got 1.05x from the first commit and 1.02x from the second; I haven't timed the whole PR against main on it. The column-scale commit also speeds up small compact-scale backward shapes (dGeLU 993x7168 1.18x). The one cell I saw go the other way is compact dbias + dSiLU at 993x7168: same speed by the timing above, 2% slower by kernel time alone. The three follow-up commits were also checked on a Rubin GPU, where every row was at parity or faster.
This doesn't depend on #3840. Both touch
CuTeDSL/activations.pyandquantize_mxfp8_common.py, so whichever merges second resolves those. #3853 (register-resident gated kernel) builds on this one.Fixes # (issue)
Type of change
Changes
util/activation_table.cuh,activation/activation_table.cu: the tables, built once per device on first usecast/dispatch/quantize.cuh: pick the table for BF16 input whenNVTE_ACTIVATION_TABLE=1;quantize_mxfp8.cuhcomputes the table values on the CUDA pathquantize_mxfp8_cutedsl.cuh: pass the table layout (part of the kernel key) and pointer to CuTeDSLCuTeDSL/activations.py: table readers, the computed table spec,act_silu = x * sigmoid(x), the fast-reciprocal sigmoidquantize_mxfp8_g2r_bidim.py: reads the tables, the new cases above, reducer-written scales, unclustered table launch, fast-reciprocal SiLUquantize_mxfp8_g2s_general.py,quantize_mxfp8_g2r_rowwise_2lane.py: compute the table values; dbias in CUDA's row ordertest_activation_table.cu(every BF16 input vs the computed activation) andtest_cast_mxfp8_act_exact.cu(fused quantize bit for bit vs per-element activation then quantize, both backends, tables on/off, BF16/FP16/FP32, dbias, tails, special values); the PyTorch and JAX CuTeDSL backend tests now also pass with tables on;test_silu_fast_reciprocal_rowscovers the SiLU fallbackdocs/envvars.rst: the two new env varsTested on GB300 with CuTeDSL on and off (act_exact, activation table, fused cast/gated C++ tests, PyTorch CuTeDSL backend, requantize and dequantize tests, racecheck/synccheck): all pass. Not run: the JAX tests (no JAX in my container), SM 10.0.
Checklist: