flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward¶
- flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward(g: Tensor, x: Tensor, w: Tensor, rstd: Tensor, *, g_residual: Tensor | None = None, workspace: Tensor | None = None, dx: Tensor | None = None, dw: Tensor | None = None) tuple[Tensor, Tensor]¶
Fused BF16 RMSNorm backward:
dxand the FP32 weight gradientdw.The call runs, per window of
T, the form the “Program selection by token count” notes list for that window: in the windows that ship it, one kernel that writesdxand adds thedwsums of its CTAs into FP64 sums in the workspace, which the CTA that finishes last rounds intodw(dxis bitwise reproducible and within one BF16 ulp ofmax(|dx|, rms(dx))of the deterministic kernel’sdx, not always bitwise equal to it; the order of the FP64 additions depends on scheduling, sodwis not guaranteed bitwise reproducible); in the others, the deterministic package’s backward pair rendered into this package.While
torch.use_deterministic_algorithms(True)is in effect every call raisesRuntimeErrornamingflashinfer.cake_rmsnorm_det_train.cake_rmsnorm_det_train_backward(), the deterministic package’s backward (dwthrough a fixed row-chunk partition and a fixed-order reduction, bitwise reproducible for a given token count on devices with the same SM count), whatever window form it would run; withwarn_only=Trueit warns once and runs this package’s backward (never the deterministic package’s).- Parameters:
g (torch.Tensor) – BF16
[T, H]gradient ofyof any layout. Views with a contiguous feature dimension, 16-byte aligned storage and a row stride that is a multiple of 16 bytes (expanded or sliced autograd gradients included) are read in place; other layouts are materialized once. An N-D gradient,[..., H], takesx’s shape and collapses its leading dimensions into rows under the forward’s rule.x (torch.Tensor) – The BF16 tensor that was normalized (
xof the plain forward,h_newof the residual forward); same layout rules asg.w (torch.Tensor) – The forward’s weight: BF16
[H](read in place when contiguous and 16-byte aligned), or, on the FP32-weight routes, FP32[H](in place when contiguous and 32-byte aligned); any other layout is copied once.rstd (torch.Tensor) – FP32
[T]reciprocal RMS returned by the forward (of the leading shape[...]for an N-D call, or flat[prod(...)]).g_residual (torch.Tensor, optional) – BF16 gradient of
g’s shape flowing directly throughh_new(residual forward), any layout. It is added once so that the returneddxis the gradient of both residual-add inputs.workspace (torch.Tensor, optional) – Caller-owned
uint8buffer fromcake_rmsnorm_nondet_train_backward_workspace(), at least as large as that helper returns for this call (one buffer serves calls of any hidden size, residual flag and weight dtype it is large enough for; seecake_rmsnorm_nondet_train_backward_workspace_bytes()). The call issues no memset, copy or synchronization on it. When omitted, the call allocates and zero-fills one: a caching-allocator call, one fill kernel (about 3 us on B200) and about 35 us of host time per call. One buffer per backward that may run concurrently (on another stream).dx (torch.Tensor, optional) – Preallocated BF16
dxofx’s shape. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch. Its previous contents are ignored.dw (torch.Tensor, optional) – Preallocated FP32
[H]weight gradient. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch. Its previous contents are ignored.
- Returns:
(dx, dw)withdwin FP32; the caller decides whether to cast it.T == 0launches nothing and returns an emptydxand a zerodw.- Return type:
tuple[torch.Tensor, torch.Tensor]
Notes
The program windows of every architecture and route, and the program each window runs, are listed in the docstring of
flashinfer.experimental.cake.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward().