flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det

flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det(x: Tensor, w: Tensor, offs: Tensor, *, fuse_wgrad_accumulation: bool = False) → Tensor

Differentiable ragged grouped GEMM Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T.

The slice lower bound is 0 for e == 0 and offs[e-1] for e > 0. For m_indptr input, offs in these formulas means m_indptr[1:].

Parameters:
  • x (torch.Tensor) – [sum_m, K] bfloat16 rows of all groups, group after group.

  • w (torch.Tensor) – [E, N, K] bfloat16 weights ([N, K] per group).

  • offs (torch.Tensor) – int32 device tensor: [E] cumulative end offsets or m_indptr [E + 1]. Read by the kernels, never on the host.

  • fuse_wgrad_accumulation (bool) – Keyword-only. False (default): the backward returns w’s gradient as below. True (Megatron’s gradient_accumulation_fusion): w.main_grad must be an fp32 or bf16 tensor of w’s shape on its device with unit stride along the last dimension and rows and experts that do not overlap (ValueError, checked at the forward call when autograd records it: w requires grad, grad mode is on and the call is not a checkpoint recompute inside a backward; and again at the backward, which reads w.main_grad then and also refuses one that shares memory with the operands). Under torch.no_grad() or inference mode the forward needs no main_grad. The backward adds the deterministic dW into it in place (main_grad[e] += G[e].T @ X[e], the package’s accumulate mode: fused into the weight-gradient programs where the route serves their accumulate builds for main_grad’s dtype, else dW in main_grad’s dtype into a scratch, then main_grad.add_ on the same stream), which gives AccumulateGrad’s bits of that dW on either route, bitwise reproducible run to run, and returns for w a fresh torch.empty of w’s shape and dtype when w has the attribute grad_added_to_main_grad (set to True; torch.zeros when w.zero_out_wgrad is true), else None: w.grad is then untouched and no gradient hook of w fires. Every backward adds (retain_graph twice adds twice), also a torch.autograd.grad whose inputs exclude w (split backward-weight schedules are not supported); a w that does not require grad is never added to. Not supported (NotImplementedError) under torch.jit.trace, torch.compile and a create_graph backward; under a reentrant checkpoint w arrives detached, without main_grad (ValueError). Cost: on the composed route the weight-gradient launch holds a scratch of E * N * K elements of main_grad’s dtype (4 * E * N * K bytes for fp32, freed after the add), which the programs write and the add reads back besides its read and write of main_grad; the fused route holds none. The placeholder takes 2 * E * N * K bytes. Under torch.use_deterministic_algorithms(True) with torch.utils.deterministic.fill_uninitialized_memory on (the default) torch also fills the scratch and the torch.empty placeholder with NaN when it allocates them.

Returns:

[sum_m, N] bfloat16. backward produces x.grad [sum_m, K] bfloat16 and w.grad [E, N, K] in w’s dtype (bfloat16); use cake_moe_grouped_gemm_det_train_wgrad() directly for an fp32 weight gradient, or fuse_wgrad_accumulation=True to add it into an fp32 main_grad. The forward GEMM runs after autograd saved x, w and offs: under a non-reentrant checkpoint with early stop on, the recomputation skips it when these are the last saves of the innermost checkpointed region, as for a built-in op.

Return type:

torch.Tensor

Notes

Deterministic only: the forward and both gradients reduce in a fixed order and are bitwise reproducible run to run, with or without fuse_wgrad_accumulation. There is no deterministic parameter (passing one raises TypeError) and torch.use_deterministic_algorithms(True) changes nothing (no alert, no warning). The non-deterministic wrapper is flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet.