flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward

flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward(x: Tensor, w: Tensor, eps: float, *, residual: Tensor | None = None, out: Tensor | None = None, rstd: Tensor | None = None, residual_out: Tensor | None = None) → tuple[Tensor, ...]

Fused BF16 RMSNorm forward with FP32 statistics.

This package’s forward programs: the deterministic package’s programs rendered here (the forward has no cross-row reduction, so its outputs are bitwise reproducible). torch.use_deterministic_algorithms does not affect the forward.

Parameters:
  • x (torch.Tensor) – BF16 [T, H] input of any layout. Row-strided views with a contiguous feature dimension, 32-byte aligned storage and a row stride that is a multiple of 32 bytes are read in place; other layouts (odd column offset, odd row stride, transposed storage) are materialized once. A tensor of more dimensions, [..., H], is normalized row by row when its feature dimension is contiguous and its leading dimensions are contiguous with one another in row-major order (stride(i) == stride(i + 1) * size(i + 1); a [T, heads, H] slice of a wider [T, heads, W] buffer qualifies): the leading dimensions collapse into prod(...) rows, the outputs take the input’s leading shape (rstd is [...]), w is shared by every row and the backward’s dw sums them all; any other N-D layout raises ValueError naming the rule. With residual this is the residual stream h and the normalized value is h_new = BF16(h + residual).

  • w (torch.Tensor) – BF16 [H] weight, read in place when contiguous and 32-byte aligned, else copied once. An FP32 [H] weight – the effective weight 1 + w a zero-centred model computes in FP32 – is accepted on the routes that ship the FP32-weight programs (cake_jit.weight_dtypes), else ValueError.

  • eps (float) – Variance epsilon (FP32 inside the kernel).

  • residual (torch.Tensor, optional) – BF16 update of x’s shape added to x before normalization (same layout rules as x). Enables the fused residual-add route.

  • out (torch.Tensor, optional) – Preallocated BF16 y of x’s shape. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.

  • rstd (torch.Tensor, optional) – Preallocated FP32 reciprocal RMS of x’s leading shape ([...] for x of shape [..., H]). Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.

  • residual_out (torch.Tensor, optional) – Preallocated BF16 h_new of x’s shape, used only with residual. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch.

Returns:

(y, rstd) or (y, rstd, h_new) when residual is given. rstd is the FP32 per-row 1 / sqrt(mean(h_new^2) + eps) the backward needs; no other [T, H] temporary is produced.

Return type:

tuple[torch.Tensor, …]

Notes

The program windows of every architecture and route, and the program each window runs, are listed in the docstring of flashinfer.experimental.cake.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_forward().