flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet_train_wgrad

flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet_train_wgrad(g: Tensor, x: Tensor, offs: Tensor, out: Tensor | None = None, out_dtype: dtype | None = None, num_groups: int | None = None, *, accumulate: bool = False) → Tensor

dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]] for every group e.

The slice lower bound is 0 for e == 0 and offs[e-1] for e > 0. For m_indptr input, offs in these formulas means m_indptr[1:].

g is [sum_m, N] bfloat16, x is [sum_m, K] bfloat16, offs is [E] int32 device end offsets (E = offs.numel(); pass num_groups=E to hand in m_indptr [E + 1] instead) and out is [E, N, K] in out_dtype (bfloat16 or float32; defaults to the dtype of out, else bfloat16). fp32 accumulation, one rounding at the store, exact zeros for empty groups. With the 512-wide k tile a split tail is summed in slot order in the same launch, so dW repeats bit for bit for the same inputs, shapes, device SM count and configuration, and no launch alerts under torch.use_deterministic_algorithms(True). N % 256 == 0 and K a multiple of the selected k tile (256, or 512 when chosen) are required. See cake_moe_grouped_gemm_nondet_train_forward for the prepared form.

accumulate=True adds into out instead, out[e] += G[e].T @ X[e] (Megatron’s gradient-accumulation fusion; out required, bf16 or fp32, out_dtype None or its dtype), and returns out: fused into the 512-wide k tile’s epilogue where the route serves that stage’s accumulate build, else the overwrite into a scratch followed by out.add_ (cake_moe_grouped_gemm_nondet_train_prepare_wgrad and the package README give the routes and the numerical contract).

Parameters:
  • g (torch.Tensor) – BF16 [sum_m, N] packed output-gradient rows.

  • x (torch.Tensor) – BF16 [sum_m, K] packed activation rows.

  • offs (torch.Tensor) – int32 device cumulative end offsets [E], or leading-zero m_indptr [E + 1].

  • out (torch.Tensor, optional) – Caller-owned [E, N, K] output; required when accumulating.

  • out_dtype (torch.dtype, optional) – torch.bfloat16 or torch.float32; defaults to out.dtype or bfloat16 without out.

  • num_groups (int, optional) – Expert count E; must be supplied for m_indptr input.

  • accumulate (bool, default=False) – Add the weight gradient into out instead of overwriting it.