flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet

flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet(x: Tensor, w: Tensor, offs: Tensor, *, fuse_wgrad_accumulation: bool = False) → Tensor

Differentiable ragged grouped GEMM Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T.

The slice lower bound is 0 for e == 0 and offs[e-1] for e > 0. For m_indptr input, offs in these formulas means m_indptr[1:].

Parameters:
  • x (torch.Tensor) – [sum_m, K] bfloat16 rows of all groups, group after group.

  • w (torch.Tensor) – [E, N, K] bfloat16 weights ([N, K] per group).

  • offs (torch.Tensor) – int32 device tensor: [E] cumulative end offsets or m_indptr [E + 1]. Read by the kernels, never on the host.

  • fuse_wgrad_accumulation (bool) – False (default): the backward returns w’s gradient as below. True (Megatron’s gradient_accumulation_fusion): w.main_grad must be an fp32 or bf16 tensor of w’s shape on its device with unit stride along the last dimension (ValueError, checked at the forward call when autograd records it: w requires grad, grad mode is on and the call is not a checkpoint recompute inside a backward; and again at the backward, which reads w.main_grad then). Under torch.no_grad() or inference mode the forward needs no main_grad. The backward adds dW into it in place (main_grad[e] += G[e].T @ X[e], the package’s accumulate mode, fused into the weight-gradient kernel where the route serves it) and returns for w a fresh torch.empty of w’s shape and dtype when w has the attribute grad_added_to_main_grad (set to True; torch.zeros when w.zero_out_wgrad is true), else None: w.grad is then untouched and no gradient hook of w fires. Every backward adds (retain_graph twice adds twice), also a torch.autograd.grad whose inputs exclude w (split backward-weight schedules are not supported); a w that does not require grad is never added to. Not supported (NotImplementedError) under torch.jit.trace, torch.compile and a create_graph backward; under a reentrant checkpoint w arrives detached, without main_grad (ValueError).

Returns:

[sum_m, N] bfloat16, the deterministic forward’s bits. backward produces x.grad [sum_m, K] bfloat16 (the deterministic bits) and w.grad [E, N, K] in w’s dtype (bfloat16), which repeats bit for bit for the same inputs, shapes, device SM count and configuration (a split tail of the 512-wide k tile is summed in slot order); use cake_moe_grouped_gemm_nondet_train_wgrad() directly for an fp32 weight gradient. No launch of the forward or the backward alerts under torch.use_deterministic_algorithms(True); an order-varying weight-gradient launch, which only an architecture listed in unordered_tail_archs plans, would raise the non-determinism alert immediately before it runs (RuntimeError; under warn_only=True a UserWarning, then it runs; no fallback). The forward GEMM runs after autograd saved x, w and offs: under a non-reentrant checkpoint with early stop on, the recomputation skips it when these are the last saves of the innermost checkpointed region, as for a built-in op.

Return type:

torch.Tensor

Notes

Non-deterministic only: there is no deterministic parameter (passing one raises TypeError). The deterministic wrapper is flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det; nothing here routes to it.