flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_wgrad¶
- flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_wgrad(g: Tensor, x: Tensor, offs: Tensor, out: Tensor | None = None, out_dtype: dtype | None = None, num_groups: int | None = None, *, accumulate: bool = False) Tensor¶
dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]]for every groupe.The slice lower bound is
0fore == 0andoffs[e-1]fore > 0. Form_indptrinput,offsin these formulas meansm_indptr[1:].gis[sum_m, N]bfloat16,xis[sum_m, K]bfloat16,offsis[E]int32 device end offsets (E = offs.numel(); passnum_groups=Eto hand inm_indptr[E + 1]instead) andoutis[E, N, K]inout_dtype(bfloat16 or float32; defaults to the dtype ofout, else bfloat16). fp32 accumulation, bitwise deterministic, exact zeros for empty groups.N % 256 == 0andKa multiple of the selected k tile (256, or 512 when chosen) are required. Seecake_moe_grouped_gemm_det_train_forwardfor the prepared form.accumulate=True(keyword-only, defaultFalse) adds intooutinstead,out[e] += G[e].T @ X[e](Megatron’s gradient-accumulation fusion into an fp32 or bf16main_grad;outrequired, bfloat16 or float32,out_dtypeNone or its dtype, no overlap ofoutwith itself,g,xoroffs:ValueError), and returnsout, which holds AccumulateGrad’s bits of the deterministic weight gradient in its dtype, bitwise reproducible run to run: fused into the 512-wide k tile’s programs (the main stage and the ordered tail reduce add intooutin place) where the route serves their accumulate builds forout’s dtype, else the overwrite into a scratch ofout’s dtype (E * N * Kelements,4 * E * N * Kbytes for fp32) added without.add_on the same stream; both give the same bits (cake_moe_grouped_gemm_det_train_prepare_wgradand the package README give the routes; a prepared launch is CUDA-graph capturable and every launch and replay adds again).sum_m == 0leavesoutunchanged.- Parameters:
g (torch.Tensor) – BF16
[sum_m, N]packed output-gradient rows.x (torch.Tensor) – BF16
[sum_m, K]packed activation rows.offs (torch.Tensor) – int32 device cumulative end offsets
[E], or leading-zerom_indptr [E + 1].out (torch.Tensor, optional) – Caller-owned
[E, N, K]output; required when accumulating.out_dtype (torch.dtype, optional) –
torch.bfloat16ortorch.float32; defaults toout.dtypeor bfloat16 withoutout.num_groups (int, optional) – Expert count
E; must be supplied form_indptrinput.accumulate (bool, default=False) – Add the weight gradient into
outinstead of overwriting it.