flashinfer.cake_rmsnorm_silu_gate_nondet_train.cake_rmsnorm_silu_gate_nondet¶
- flashinfer.cake_rmsnorm_silu_gate_nondet_train.cake_rmsnorm_silu_gate_nondet(x: Tensor, z: Tensor, weight: Tensor, eps: float = 1e-06) Tensor¶
Differentiable
RMSNorm(x) * weight * SiLU(z)over the last dimension (128) on the non-deterministic kernels: aCakeRMSNormSiLUGateNondetFunctionnode (the backend package’s class), which saves onlyx,zandweight.Every output of the node repeats bit for bit but a BF16 weight gradient in the fused band, whose last bits may vary from run to run; under
torch.use_deterministic_algorithms(True)the backward alerts immediately before computing the weight gradient in such a band, and nothing else alerts. A non-reentrant checkpoint replays the forward through this function (the samey). The deterministic one isflashinfer.cake_rmsnorm_silu_gate_det_train.cake_rmsnorm_silu_gate_det().- Parameters:
x (torch.Tensor) – BF16
[..., 128]input ([T, H, 128]or[T * H, 128]).z (torch.Tensor) – BF16 gate input of the same shape as
x.weight (torch.Tensor) – BF16
[128]weight shared by every token and head.eps (float) – Variance epsilon (FP32 inside the kernel).
- Returns:
BF16
yofx’s shape.- Return type:
torch.Tensor