flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet¶
- flashinfer.cake_moe_grouped_gemm_nondet_train.cake_moe_grouped_gemm_nondet(x: Tensor, w: Tensor, offs: Tensor, *, fuse_wgrad_accumulation: bool = False) Tensor¶
Differentiable ragged grouped GEMM
Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T.The slice lower bound is
0fore == 0andoffs[e-1]fore > 0. Form_indptrinput,offsin these formulas meansm_indptr[1:].- Parameters:
x (torch.Tensor) –
[sum_m, K]bfloat16 rows of all groups, group after group.w (torch.Tensor) –
[E, N, K]bfloat16 weights ([N, K]per group).offs (torch.Tensor) – int32 device tensor:
[E]cumulative end offsets orm_indptr[E + 1]. Read by the kernels, never on the host.fuse_wgrad_accumulation (bool) –
False(default): the backward returnsw’s gradient as below.True(Megatron’sgradient_accumulation_fusion):w.main_gradmust be an fp32 or bf16 tensor ofw’s shape on its device with unit stride along the last dimension (ValueError, checked at the forward call when autograd records it:wrequires grad, grad mode is on and the call is not a checkpoint recompute inside a backward; and again at the backward, which readsw.main_gradthen). Undertorch.no_grad()or inference mode the forward needs nomain_grad. The backward addsdWinto it in place (main_grad[e] += G[e].T @ X[e], the package’s accumulate mode, fused into the weight-gradient kernel where the route serves it) and returns forwa freshtorch.emptyofw’s shape and dtype whenwhas the attributegrad_added_to_main_grad(set toTrue;torch.zeroswhenw.zero_out_wgradis true), elseNone:w.gradis then untouched and no gradient hook ofwfires. Every backward adds (retain_graphtwice adds twice), also atorch.autograd.gradwhoseinputsexcludew(split backward-weight schedules are not supported); awthat does not require grad is never added to. Not supported (NotImplementedError) undertorch.jit.trace,torch.compileand acreate_graphbackward; under a reentrant checkpointwarrives detached, withoutmain_grad(ValueError).
- Returns:
[sum_m, N]bfloat16, the deterministic forward’s bits.backwardproducesx.grad[sum_m, K]bfloat16 (the deterministic bits) andw.grad[E, N, K]inw’s dtype (bfloat16), which repeats bit for bit for the same inputs, shapes, device SM count and configuration (a split tail of the 512-wide k tile is summed in slot order); usecake_moe_grouped_gemm_nondet_train_wgrad()directly for an fp32 weight gradient. No launch of the forward or the backward alerts undertorch.use_deterministic_algorithms(True); an order-varying weight-gradient launch, which only an architecture listed inunordered_tail_archsplans, would raise the non-determinism alert immediately before it runs (RuntimeError; underwarn_only=TrueaUserWarning, then it runs; no fallback). The forward GEMM runs after autograd savedx,wandoffs: under a non-reentrant checkpoint with early stop on, the recomputation skips it when these are the last saves of the innermost checkpointed region, as for a built-in op.- Return type:
torch.Tensor
Notes
Non-deterministic only: there is no
deterministicparameter (passing one raisesTypeError). The deterministic wrapper isflashinfer.cake_moe_grouped_gemm_det_train.cake_moe_grouped_gemm_det; nothing here routes to it.