flashinfer.cake_lm_head_nondet_train¶
cake_lm_head_nondet_train holds the non-deterministic Cake training entries of the chunked
LM-head projection with a cross-entropy or policy loss, its per-token log-probabilities, and the
log-probabilities / KL over a ragged candidate support, on SM100 (B200 / GB200), SM103
(B300 / GB300) and SM107 (R200). The entries take the arguments of the deterministic module
flashinfer.cake_lm_head_det_train and keep its precision boundaries. Their bits may differ
from the deterministic module’s, within the accuracy budgets, where the default path forms an
output in another order: a sliced dX GEMM stores its K slices into an FP32 slab and a slab sum
adds them in slice order (dX); zero-weight policy rows and zero-dlogp log-probability
rows skip the gradient GEMMs (loss / dW); a call of fewer than three chunks folds a short
last chunk into the chunk before it (loss / dW); a one-row cross-entropy loss call at
H = 6144 runs one-pass programs (loss / logp / lse / dW); a log-probability
backward of 2 to 4 chunks runs vocabulary-major by the README’s rule (dX / dW); and the
candidate-support dense dX uses a wider vocabulary block (dX). No entry takes a
deterministic keyword.
The module makes no run-to-run bitwise promise for the gradients, and keeps PyTorch’s alert as
part of that contract: under torch.use_deterministic_algorithms(True) an entry raises the
alert right before a K-sliced dX reduction and never falls back to the deterministic module.
The backend package is flashinfer.experimental.cake.cake_lm_head_nondet_train; its README
lists the output differences.
|
Differentiable chunked LM-head projection + loss with a memory-bounded backward and a slab-summed dX reduction, SM100 / SM103 / SM107 only. |
|
Differentiable chunked selected-token log-probability with a slab-summed dX reduction, SM100 / SM103 / SM107 only. |
|
Do part of the one-time work of a process's first chunked LM-head call ahead of it, then wait for the device once (SM100 / SM103 / SM107 only). |
Differentiable log-probabilities over a ragged candidate support per token, SM100 / SM103 / SM107 only. |
|
|
Unreduced forward KL from a detached teacher over a ragged candidate support, SM100 / SM103 / SM107 only. |