flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det¶
- flashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det(x: Tensor, w: Tensor, offs: Tensor, *, fuse_wgrad_accumulation: bool = False) Tensor¶
Differentiable ragged grouped GEMM
Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T.The slice lower bound is
0fore == 0andoffs[e-1]fore > 0. Form_indptrinput,offsin these formulas meansm_indptr[1:].- Parameters:
x (torch.Tensor) –
[sum_m, K]bfloat16 rows of all groups, group after group.w (torch.Tensor) –
[E, N, K]bfloat16 weights ([N, K]per group).offs (torch.Tensor) – int32 device tensor:
[E]cumulative end offsets orm_indptr[E + 1]. Read by the kernels, never on the host.fuse_wgrad_accumulation (bool) – Keyword-only.
False(default): the backward returnsw’s gradient as below.True(Megatron’sgradient_accumulation_fusion):w.main_gradmust be an fp32 or bf16 tensor ofw’s shape on its device with unit stride along the last dimension and rows and experts that do not overlap (ValueError, checked at the forward call when autograd records it:wrequires grad, grad mode is on and the call is not a checkpoint recompute inside a backward; and again at the backward, which readsw.main_gradthen and also refuses one that shares memory with the operands). Undertorch.no_grad()or inference mode the forward needs nomain_grad. The backward adds the deterministicdWinto it in place (main_grad[e] += G[e].T @ X[e], the package’s accumulate mode: fused into the weight-gradient programs where the route serves their accumulate builds formain_grad’s dtype, elsedWinmain_grad’s dtype into a scratch, thenmain_grad.add_on the same stream), which gives AccumulateGrad’s bits of thatdWon either route, bitwise reproducible run to run, and returns forwa freshtorch.emptyofw’s shape and dtype whenwhas the attributegrad_added_to_main_grad(set toTrue;torch.zeroswhenw.zero_out_wgradis true), elseNone:w.gradis then untouched and no gradient hook ofwfires. Every backward adds (retain_graphtwice adds twice), also atorch.autograd.gradwhoseinputsexcludew(split backward-weight schedules are not supported); awthat does not require grad is never added to. Not supported (NotImplementedError) undertorch.jit.trace,torch.compileand acreate_graphbackward; under a reentrant checkpointwarrives detached, withoutmain_grad(ValueError). Cost: on the composed route the weight-gradient launch holds a scratch ofE * N * Kelements ofmain_grad’s dtype (4 * E * N * Kbytes for fp32, freed after the add), which the programs write and the add reads back besides its read and write ofmain_grad; the fused route holds none. The placeholder takes2 * E * N * Kbytes. Undertorch.use_deterministic_algorithms(True)withtorch.utils.deterministic.fill_uninitialized_memoryon (the default) torch also fills the scratch and thetorch.emptyplaceholder with NaN when it allocates them.
- Returns:
[sum_m, N]bfloat16.backwardproducesx.grad[sum_m, K]bfloat16 andw.grad[E, N, K]inw’s dtype (bfloat16); usecake_moe_grouped_gemm_det_train_wgrad()directly for an fp32 weight gradient, orfuse_wgrad_accumulation=Trueto add it into an fp32main_grad. The forward GEMM runs after autograd savedx,wandoffs: under a non-reentrant checkpoint with early stop on, the recomputation skips it when these are the last saves of the innermost checkpointed region, as for a built-in op.- Return type:
torch.Tensor
Notes
Deterministic only: the forward and both gradients reduce in a fixed order and are bitwise reproducible run to run, with or without
fuse_wgrad_accumulation. There is nodeterministicparameter (passing one raisesTypeError) andtorch.use_deterministic_algorithms(True)changes nothing (no alert, no warning). The non-deterministic wrapper isflashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet.