flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward

flashinfer.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward(g: Tensor, x: Tensor, w: Tensor, rstd: Tensor, *, g_residual: Tensor | None = None, workspace: Tensor | None = None, dx: Tensor | None = None, dw: Tensor | None = None) → tuple[Tensor, Tensor]

Fused BF16 RMSNorm backward: dx and the FP32 weight gradient dw.

The call runs, per window of T, the form the “Program selection by token count” notes list for that window: in the windows that ship it, one kernel that writes dx and adds the dw sums of its CTAs into FP64 sums in the workspace, which the CTA that finishes last rounds into dw (dx is bitwise reproducible and within one BF16 ulp of max(|dx|, rms(dx)) of the deterministic kernel’s dx, not always bitwise equal to it; the order of the FP64 additions depends on scheduling, so dw is not guaranteed bitwise reproducible); in the others, the deterministic package’s backward pair rendered into this package.

While torch.use_deterministic_algorithms(True) is in effect every call raises RuntimeError naming flashinfer.cake_rmsnorm_det_train.cake_rmsnorm_det_train_backward(), the deterministic package’s backward (dw through a fixed row-chunk partition and a fixed-order reduction, bitwise reproducible for a given token count on devices with the same SM count), whatever window form it would run; with warn_only=True it warns once and runs this package’s backward (never the deterministic package’s).

Parameters:
  • g (torch.Tensor) – BF16 [T, H] gradient of y of any layout. Views with a contiguous feature dimension, 16-byte aligned storage and a row stride that is a multiple of 16 bytes (expanded or sliced autograd gradients included) are read in place; other layouts are materialized once. An N-D gradient, [..., H], takes x’s shape and collapses its leading dimensions into rows under the forward’s rule.

  • x (torch.Tensor) – The BF16 tensor that was normalized (x of the plain forward, h_new of the residual forward); same layout rules as g.

  • w (torch.Tensor) – The forward’s weight: BF16 [H] (read in place when contiguous and 16-byte aligned), or, on the FP32-weight routes, FP32 [H] (in place when contiguous and 32-byte aligned); any other layout is copied once.

  • rstd (torch.Tensor) – FP32 [T] reciprocal RMS returned by the forward (of the leading shape [...] for an N-D call, or flat [prod(...)]).

  • g_residual (torch.Tensor, optional) – BF16 gradient of g’s shape flowing directly through h_new (residual forward), any layout. It is added once so that the returned dx is the gradient of both residual-add inputs.

  • workspace (torch.Tensor, optional) – Caller-owned uint8 buffer from cake_rmsnorm_nondet_train_backward_workspace(), at least as large as that helper returns for this call (one buffer serves calls of any hidden size, residual flag and weight dtype it is large enough for; see cake_rmsnorm_nondet_train_backward_workspace_bytes()). The call issues no memset, copy or synchronization on it. When omitted, the call allocates and zero-fills one: a caching-allocator call, one fill kernel (about 3 us on B200) and about 35 us of host time per call. One buffer per backward that may run concurrently (on another stream).

  • dx (torch.Tensor, optional) – Preallocated BF16 dx of x’s shape. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch. Its previous contents are ignored.

  • dw (torch.Tensor, optional) – Preallocated FP32 [H] weight gradient. Contiguous aligned buffers are written directly; any other layout receives a copy after the launch. Its previous contents are ignored.

Returns:

(dx, dw) with dw in FP32; the caller decides whether to cast it. T == 0 launches nothing and returns an empty dx and a zero dw.

Return type:

tuple[torch.Tensor, torch.Tensor]

Notes

The program windows of every architecture and route, and the program each window runs, are listed in the docstring of flashinfer.experimental.cake.cake_rmsnorm_nondet_train.cake_rmsnorm_nondet_train_backward().