flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_prepare_wgrad

flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det_train_prepare_wgrad(g: torch.Tensor, x: torch.Tensor, offs: torch.Tensor, out: torch.Tensor | None = None, out_dtype: torch.dtype | None = None, num_groups: int | None = None, *, accumulate: bool = False) → CakeMoEGroupedGemmDetLaunch

Prepare dW[E, N, K] = grouped G[e].T @ X[e] (bf16 in; bf16 or fp32 out), deterministic.

g: [sum_m, N] bf16; x: [sum_m, K] bf16; offs: int32 [E] device end offsets (E = offs.numel() unless num_groups is given, in which case an m_indptr of E + 1 entries is accepted too); N % 256 == 0 and K a multiple of the selected k tile (256, or 512 when it is chosen). Empty groups produce exact zeros. out_dtype defaults to the dtype of out (bf16 when neither is given).

accumulate=True (keyword-only, default False) adds instead: every launch computes out[e] <- out[e] + G[e].T @ X[e] into the caller’s out (required, bf16 or fp32; out_dtype None or out.dtype; rows and experts that do not overlap, no memory shared with g, x or offs; ValueError otherwise, before anything is allocated or bound), so out holds AccumulateGrad’s bits of the deterministic weight gradient in its dtype, bitwise reproducible run to run (an empty group adds exact zeros: its values stay, -0.0 becomes +0.0). plan["accumulate"] is "fused" (the accumulate builds of the main stage and, for a split tail, of the tail reduce add into out in place: stages is the overwrite’s with ACC_SUFFIX on each) or "composed" (the overwrite of this plan into a contiguous scratch of out.dtype, E * N * K elements that this preparation allocates and the launch owns, then out.add_(scratch) on the same stream; plan["accumulate_reason"] says why); both give the same bits, and every other plan key is the overwrite’s (module docstring, accumulate mode). No row (sum_m == 0) launches nothing and leaves out unchanged. Each launch() and each replay of a graph that captured it adds again; keep the launch alive while such a graph may replay.

Parameters:
  • g (torch.Tensor) – BF16 [sum_m, N] packed output-gradient rows.

  • x (torch.Tensor) – BF16 [sum_m, K] packed activation rows.

  • offs (torch.Tensor) – int32 device cumulative end offsets [E], or leading-zero m_indptr [E + 1].

  • out (torch.Tensor, optional) – Caller-owned [E, N, K] output; required when accumulating.

  • out_dtype (torch.dtype, optional) – torch.bfloat16 or torch.float32; defaults to out.dtype or bfloat16 without out.

  • num_groups (int, optional) – Expert count E; must be supplied for m_indptr input.

  • accumulate (bool, default=False) – Add the weight gradient into out instead of overwriting it.