flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_wgrad

flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_wgrad(g: Tensor, x: Tensor, offs: Tensor, out: Tensor | None = None, out_dtype: dtype | None = None, num_groups: int | None = None, *, accumulate: bool = False) → Tensor

dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]] for every group e.

The slice lower bound is 0 for e == 0 and offs[e-1] for e > 0. For m_indptr input, offs in these formulas means m_indptr[1:].

g is [sum_m, N] bfloat16, x is [sum_m, K] bfloat16, offs is [E] int32 device end offsets (E = offs.numel(); pass num_groups=E to hand in m_indptr [E + 1] instead) and out is [E, N, K] in out_dtype (bfloat16 or float32; defaults to the dtype of out, else bfloat16). fp32 accumulation, bitwise deterministic, exact zeros for empty groups. N % 256 == 0 and K a multiple of the selected k tile (256, or 512 when chosen) are required. See cake_moe_grouped_gemm_det_train_forward for the prepared form.

accumulate=True (keyword-only, default False) adds into out instead, out[e] += G[e].T @ X[e] (Megatron’s gradient-accumulation fusion into an fp32 or bf16 main_grad; out required, bfloat16 or float32, out_dtype None or its dtype, no overlap of out with itself, g, x or offs: ValueError), and returns out, which holds AccumulateGrad’s bits of the deterministic weight gradient in its dtype, bitwise reproducible run to run: fused into the 512-wide k tile’s programs (the main stage and the ordered tail reduce add into out in place) where the route serves their accumulate builds for out’s dtype, else the overwrite into a scratch of out’s dtype (E * N * K elements, 4 * E * N * K bytes for fp32) added with out.add_ on the same stream; both give the same bits (cake_moe_grouped_gemm_det_train_prepare_wgrad and the package README give the routes; a prepared launch is CUDA-graph capturable and every launch and replay adds again). sum_m == 0 leaves out unchanged.

Parameters:
  • g (torch.Tensor) – BF16 [sum_m, N] packed output-gradient rows.

  • x (torch.Tensor) – BF16 [sum_m, K] packed activation rows.

  • offs (torch.Tensor) – int32 device cumulative end offsets [E], or leading-zero m_indptr [E + 1].

  • out (torch.Tensor, optional) – Caller-owned [E, N, K] output; required when accumulating.

  • out_dtype (torch.dtype, optional) – torch.bfloat16 or torch.float32; defaults to out.dtype or bfloat16 without out.

  • num_groups (int, optional) – Expert count E; must be supplied for m_indptr input.

  • accumulate (bool, default=False) – Add the weight gradient into out instead of overwriting it.