flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward¶
- flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward(x: Tensor, w: Tensor, eps: float, *, residual: Tensor | None = None, out: Tensor | None = None, rstd: Tensor | None = None, residual_out: Tensor | None = None) tuple[Tensor, ...]¶
Fused BF16 RMSNorm forward with FP32 statistics.
This package’s forward programs: the deterministic package’s programs rendered here (the forward has no cross-row reduction, so its outputs are bitwise reproducible).
torch.use_deterministic_algorithmsdoes not affect the forward.- Parameters:
x (torch.Tensor) – BF16
[T, H]input of any layout. Row-strided views with a contiguous feature dimension, 32-byte aligned storage and a row stride that is a multiple of 32 bytes are read in place; other layouts (odd column offset, odd row stride, transposed storage) are materialized once. A tensor of more dimensions,[..., H], is normalized row by row when its feature dimension is contiguous and its leading dimensions are contiguous with one another in row-major order (stride(i) == stride(i + 1) * size(i + 1); a[T, heads, H]slice of a wider[T, heads, W]buffer qualifies): the leading dimensions collapse intoprod(...)rows, the outputs take the input’s leading shape (rstdis[...]),wis shared by every row and the backward’sdwsums them all; any other N-D layout raisesValueErrornaming the rule. Withresidualthis is the residual streamhand the normalized value ish_new = BF16(h + residual).w (torch.Tensor) – BF16
[H]weight, read in place when contiguous and 32-byte aligned, else copied once. An FP32[H]weight – the effective weight1 + wa zero-centred model computes in FP32 – is accepted on the routes that ship the FP32-weight programs (cake_jit.weight_dtypes), elseValueError.eps (float) – Variance epsilon (FP32 inside the kernel).
residual (torch.Tensor, optional) – BF16 update of
x’s shape added toxbefore normalization (same layout rules asx). Enables the fused residual-add route.out (torch.Tensor, optional) – Preallocated BF16
yofx’s shape. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.rstd (torch.Tensor, optional) – Preallocated FP32 reciprocal RMS of
x’s leading shape ([...]forxof shape[..., H]). Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.residual_out (torch.Tensor, optional) – Preallocated BF16
h_newofx’s shape, used only withresidual. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.
- Returns:
(y, rstd)or(y, rstd, h_new)whenresidualis given.rstdis the FP32 per-row1 / sqrt(mean(h_new^2) + eps)the backward needs; no other[T, H]temporary is produced.- Return type:
tuple[torch.Tensor, …]
Notes
The program windows of every architecture and route, and the program each window runs, are listed in the docstring of
flashinfer.experimental.cake.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward().