flashinfer.cake_rmsnorm_silu_gate_nondet_train.cake_rmsnorm_silu_gate_nondet

flashinfer.cake_rmsnorm_silu_gate_nondet_train.cake_rmsnorm_silu_gate_nondet(x: Tensor, z: Tensor, weight: Tensor, eps: float = 1e-06) → Tensor

Differentiable RMSNorm(x) * weight * SiLU(z) over the last dimension (128) on the non-deterministic kernels: a CakeRMSNormSiLUGateNondetFunction node (the backend package’s class), which saves only x, z and weight.

Every output of the node repeats bit for bit but a BF16 weight gradient in the fused band, whose last bits may vary from run to run; under torch.use_deterministic_algorithms(True) the backward alerts immediately before computing the weight gradient in such a band, and nothing else alerts. A non-reentrant checkpoint replays the forward through this function (the same y). The deterministic one is flashinfer.cake_rmsnorm_silu_gate_det_train.cake_rmsnorm_silu_gate_det().

Parameters:
  • x (torch.Tensor) – BF16 [..., 128] input ([T, H, 128] or [T * H, 128]).

  • z (torch.Tensor) – BF16 gate input of the same shape as x.

  • weight (torch.Tensor) – BF16 [128] weight shared by every token and head.

  • eps (float) – Variance epsilon (FP32 inside the kernel).

Returns:

BF16 y of x’s shape.

Return type:

torch.Tensor