flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet_train_wgrad¶
- flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet_train_wgrad(g: Tensor, x: Tensor, offs: Tensor, out: Tensor | None = None, out_dtype: dtype | None = None, num_groups: int | None = None, *, accumulate: bool = False) Tensor¶
dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]]for every groupe.The slice lower bound is
0fore == 0andoffs[e-1]fore > 0. Form_indptrinput,offsin these formulas meansm_indptr[1:].gis[sum_m, N]bfloat16,xis[sum_m, K]bfloat16,offsis[E]int32 device end offsets (E = offs.numel(); passnum_groups=Eto hand inm_indptr[E + 1]instead) andoutis[E, N, K]inout_dtype(bfloat16 or float32; defaults to the dtype ofout, else bfloat16). fp32 accumulation, one rounding at the store, exact zeros for empty groups. With the 512-wide k tile a split tail is summed in slot order in the same launch, sodWrepeats bit for bit for the same inputs, shapes, device SM count and configuration, and no launch alerts undertorch.use_deterministic_algorithms(True).N % 256 == 0andKa multiple of the selected k tile (256, or 512 when chosen) are required. Seecake_moe_grouped_gemm_nondet_train_forwardfor the prepared form.accumulate=Trueadds intooutinstead,out[e] += G[e].T @ X[e](Megatron’s gradient-accumulation fusion;outrequired, bf16 or fp32,out_dtypeNone or its dtype), and returnsout: fused into the 512-wide k tile’s epilogue where the route serves that stage’s accumulate build, else the overwrite into a scratch followed byout.add_(cake_moe_grouped_gemm_nondet_train_prepare_wgradand the package README give the routes and the numerical contract).- Parameters:
g (torch.Tensor) – BF16
[sum_m, N]packed output-gradient rows.x (torch.Tensor) – BF16
[sum_m, K]packed activation rows.offs (torch.Tensor) – int32 device cumulative end offsets
[E], or leading-zerom_indptr [E + 1].out (torch.Tensor, optional) – Caller-owned
[E, N, K]output; required when accumulating.out_dtype (torch.dtype, optional) –
torch.bfloat16ortorch.float32; defaults toout.dtypeor bfloat16 withoutout.num_groups (int, optional) – Expert count
E; must be supplied form_indptrinput.accumulate (bool, default=False) – Add the weight gradient into
outinstead of overwriting it.