Cake deterministic ragged BF16 grouped GEMM

The module flashinfer.cake_moe_grouped_gemm_det_train holds the deterministic ragged BF16 grouped GEMM of the expert projections of Mixture-of-Experts training on SM100 (B200 / GB200), SM103 (B300 / GB300) and SM107 (R200). Its functions are stable FlashInfer APIs: each one hands off to the function of the same name in the backend package flashinfer.experimental.cake.cake_moe_grouped_gemm_det_train, which holds the generated programs. Importing the module loads none of the backend package. flashinfer binds the module’s names on first access, so for example flashinfer.cake_moe_grouped_gemm_det needs no import of the module. To use the module itself, import it: import flashinfer.cake_moe_grouped_gemm_det_train.

Groups are consecutive row ranges of the packed activations, described by an int32 device tensor: either offs[E] of cumulative end offsets or m_indptr[E + 1] with a leading zero (the torch._grouped_mm and flashinfer.grouped_mm.grouped_mm_bf16() conventions). The kernels read the offsets on the device; the host never synchronises on them, no padding is required and group sizes may be imbalanced, empty, one row or any size that is not a tile multiple. All operands are bfloat16 with fp32 tensor-core accumulation; the reductions are bitwise deterministic (no atomics) and empty groups produce exactly zero weight gradients.

The slice lower bound is 0 for e == 0 and offs[e-1] for e > 0. For m_indptr input, offs in these formulas means m_indptr[1:]. Standalone wgrad calls with m_indptr must also pass num_groups=E.

operation

computation

output

fwd

Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T

[sum_m, N] bfloat16

dgrad

dX[offs[e-1]:offs[e]] = G[offs[e-1]:offs[e]] @ W[e] (W in place)

[sum_m, K] bfloat16

wgrad

dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]]

[E, N, K] bfloat16 or float32

Mode rules

The module is the mode: every entry below is deterministic, and every public name carries det.

  • Every entry runs the deterministic programs: a fixed reduction order, bitwise reproducible run to run.

  • No entry takes a deterministic argument: passing one (True or False, e.g. cake_moe_grouped_gemm_det(x, w, offs, deterministic=False)) raises TypeError, and so does a fourth argument of CakeMoEGroupedGemmDetFunction.apply(x, w, offs) other than None (the fifth argument is fuse_wgrad_accumulation).

  • accumulate=True / fuse_wgrad_accumulation=True add the deterministic weight gradient into out / w.main_grad, bitwise reproducible run to run (below).

  • torch.use_deterministic_algorithms(True) (also with warn_only=True) changes nothing: no alert, no warning.

  • flashinfer.grouped_mm.grouped_mm_bf16(..., backend="cake") runs the deterministic forward (the forward programs of both modes give the same bits).

The non-deterministic mode is the separate module flashinfer.cake_moe_grouped_gemm_nondet_train (Cake ragged BF16 grouped GEMM, one-launch weight-gradient tail); its entries have the same shapes and carry nondet.

Stable API opt-in

The forward projection is reachable from the stable grouped GEMM API by naming the backend:

out = flashinfer.grouped_mm.grouped_mm_bf16(a, b, m_indptr, backend="cake")

It accepts the [cum_m, K] / [E, N, K] / [E + 1] arguments of the cuDNN backend, produces bfloat16 outputs only, has no tactic index and requires N % 256 == 0 and K % 64 == 0.

One-shot functions

cake_moe_grouped_gemm_det_train_forward(x, ...)

Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T for every group e.

cake_moe_grouped_gemm_det_train_dgrad(g, w, offs)

dX[offs[e-1]:offs[e]] = G[offs[e-1]:offs[e]] @ W[e] for every group e.

cake_moe_grouped_gemm_det_train_wgrad(g, x, offs)

dW[e] = G[offs[e-1]:offs[e]].T @ X[offs[e-1]:offs[e]] for every group e.

Autograd

cake_moe_grouped_gemm_det(x, w, offs, *[, ...])

Differentiable ragged grouped GEMM Y[offs[e-1]:offs[e]] = X[offs[e-1]:offs[e]] @ W[e].T.

cake_moe_grouped_gemm_det(x, w, offs, *, fuse_wgrad_accumulation=False) wraps the torch.autograd.Function CakeMoEGroupedGemmDetFunction of the backend package: its backward produces x.grad with the dgrad program and w.grad with the wgrad program in the dtype of w. The programs always reduce in a fixed order; there is no deterministic argument. Use cake_moe_grouped_gemm_det_train_wgrad() with out_dtype=torch.float32 for an fp32 weight gradient, or fuse_wgrad_accumulation=True to add it into an fp32 w.main_grad (below).

Weight-gradient accumulation

cake_moe_grouped_gemm_det_train_wgrad(..., out=out, accumulate=True) adds the deterministic weight gradient into the caller’s bf16 or fp32 out (out[e] += G[e].T @ X[e], Megatron’s gradient-accumulation fusion) and cake_moe_grouped_gemm_det(..., fuse_wgrad_accumulation=True) adds it into w.main_grad in its backward. Both are keyword-only and off by default; with the defaults nothing changes. The host plans the launch as the overwrite of the same call (route, k tile, stage, tail split and its ordered tail reduce) and adds on one of two routes (plan["accumulate"]). "fused": for an fp32 out on the 512-wide k tile’s half-release forms, where the accumulate builds of the stage and, for a split tail, of the tail reduce ship and can load, and out is 32-byte aligned, the kernels add into out in place (no scratch). "composed" otherwise (plan["accumulate_reason"]): the overwrite into a contiguous scratch of out’s dtype, then out.add_ on the same stream, outside autograd. Both routes give the same bits.

  • Numerics: out gets AccumulateGrad’s bits of the deterministic weight gradient in its dtype (fp32: one fp32 add of the fp32 dW; bf16: the bf16 dW added in bf16), bitwise reproducible run to run; the accuracy budget is the deterministic dW’s plus one rounding. An empty group adds exact zeros (out.add_ semantics: values kept, -0.0 becomes +0.0); sum_m == 0 leaves out unchanged. Torch’s deterministic mode changes nothing.

  • Checks (ValueError, before anything is allocated or bound): out given, out_dtype None or its dtype, bf16 or fp32, no overlap of out with itself, g, x or offs. Row and expert padding is never read or written.

  • Prepared launches: every launch() and every CUDA graph replay adds again (a launch bumps out._version at launch() and at capture, not at a replay). The composed scratch is owned by the prepared launch and allocated on the stream current at the preparation: keep the launch alive while a graph that captured it may replay and, after a launch on another stream, until that launch completes; do not run one prepared accumulate launch on two streams at once.

  • Autograd flag (the Megatron / Transformer Engine convention): w.main_grad (fp32 or bf16, w’s shape and device, unit stride along the last dimension, rows and experts that do not overlap) is checked at a forward call that autograd records and read again at the backward, which also refuses one sharing memory with the operands; the backward adds into it and returns a fresh placeholder for w when w has grad_added_to_main_grad (torch.zeros under zero_out_wgrad), else none. Every backward adds, also a torch.autograd.grad whose inputs exclude w. NotImplementedError under torch.jit.trace, torch.compile and a create_graph backward; a reentrant checkpoint passes w without main_grad (ValueError).

  • Cost (S = E * N * K): the fused route takes no scratch. The composed route’s scratch takes 4 * S bytes for fp32 (gate/up N 4096, K 6144: 384 MiB at E 4 to 3 GiB at E 32; down N 6144, K 2048: 192 MiB to 1.5 GiB) and 8 * S bytes of traffic (written, then read back by the add); with an fp32 main_grad the backward’s peak is 2 * S bytes above the default flag’s bf16 w.grad (with a bf16 main_grad the scratch is 2 * S bytes and the peak is equal). Under torch.use_deterministic_algorithms(True) with torch.utils.deterministic.fill_uninitialized_memory on, torch also fills the scratch and the placeholder with NaN when it allocates them.

Prepared launches

cake_moe_grouped_gemm_det_train_prepare_forward(x, ...)

Prepare Y[sum_m, N] = grouped X @ W[e].T (bf16 in, bf16 out).

cake_moe_grouped_gemm_det_train_prepare_dgrad(g, ...)

Prepare dX[sum_m, K] = grouped G @ W[e] (W read in place; bf16 in, bf16 out).

cake_moe_grouped_gemm_det_train_prepare_wgrad(g, ...)

Prepare dW[E, N, K] = grouped G[e].T @ X[e] (bf16 in; bf16 or fp32 out), deterministic.

Each one validates the arguments, binds one generated program and returns a CakeMoEGroupedGemmDetLaunch (also importable from the module) whose launch() performs no host synchronisation and is CUDA-graph capturable; the same prepared launch stays valid when new offsets or values are written into the bound tensors. A launch allocates only the stream workspaces of its programs (the scheduler and wave buffers, described in the cake_backend module docstring): the first launch on a CUDA stream allocates that stream’s buffers, later launches on the same stream reuse them and allocate nothing, and a launch during a CUDA-graph capture, or for a device other than the current one, allocates a set of its own (from the graph’s pool under capture). The backend package holds plain functions of the same names; a build that vendors the package calls them directly.

Support query

is_cake_moe_grouped_gemm_det_train_supported([...])

Whether the deterministic MoE grouped GEMM runs on device.

is_cake_moe_grouped_gemm_det_train_supported(device=None) is True when the device (default: the current CUDA device) has the programs of every operation: the forward, the data gradient and the weight gradient.

Removed names

The former names of this family were removed with no alias: importing one raises ModuleNotFoundError, ImportError or AttributeError. Every call in the table ran the deterministic package; move each one to this module. The table gives the versions that had each former name: the v0.7.3 release, the v0.7.4rc1 prerelease and main after v0.7.4rc1. “A former deterministic package” is flashinfer.experimental.cake_moe_grouped_gemm (v0.7.3; in v0.7.4rc1 an alias of the next one) or flashinfer.experimental.cake_moe_grouped_det_gemm (v0.7.4rc1, main). The former non-deterministic calls (deterministic=False of flashinfer.experimental.cake.grouped_mm and every entry of the package flashinfer.experimental.cake.cake_moe_grouped_gemm, one level lower) move to flashinfer.cake_moe_grouped_gemm_nondet_train.

Removed

Use instead

the former deterministic packages with their submodules; the module flashinfer.grouped_mm.cake (v0.7.3, v0.7.4rc1) or flashinfer.grouped_mm.cake_det (main)

flashinfer.cake_moe_grouped_gemm_det_train (backend flashinfer.experimental.cake.cake_moe_grouped_gemm_det_train, same submodule names)

grouped_gemm_fwd / _dgrad / _wgrad (v0.7.3, v0.7.4rc1) and grouped_det_gemm_fwd / _dgrad / _wgrad (v0.7.4rc1, main), from flashinfer.grouped_mm, its module above or a former deterministic package

cake_moe_grouped_gemm_det_train_forward / _dgrad / _wgrad

flashinfer.experimental.cake.grouped_mm.grouped_gemm_fwd / _dgrad / _wgrad with the default or deterministic=True (v0.7.4rc1)

cake_moe_grouped_gemm_det_train_forward / _dgrad / _wgrad

prepare_grouped_gemm_fwd / _dgrad / _wgrad (v0.7.3, v0.7.4rc1) and prepare_grouped_det_gemm_fwd / _dgrad / _wgrad (v0.7.4rc1, main), from a former deterministic package

cake_moe_grouped_gemm_det_train_prepare_forward / _prepare_dgrad / _prepare_wgrad

GroupedGemmLaunch (v0.7.3, v0.7.4rc1) and GroupedDetGemmLaunch (v0.7.4rc1, main), from a former deterministic package

CakeMoEGroupedGemmDetLaunch

cake_grouped_mm (v0.7.3, v0.7.4rc1) and cake_grouped_det_mm (v0.7.4rc1, main), from flashinfer.grouped_mm, its module above or a former deterministic package

cake_moe_grouped_gemm_det

flashinfer.experimental.cake.grouped_mm.cake_grouped_mm with the default or deterministic=True (v0.7.4rc1)

cake_moe_grouped_gemm_det

CakeGroupedMm (v0.7.3, v0.7.4rc1) and CakeGroupedDetMm (v0.7.4rc1, main), from a former deterministic package

CakeMoEGroupedGemmDetFunction

cake_backend.grouped_mm_bf16_cake of a former deterministic package (v0.7.3, v0.7.4rc1)

flashinfer.grouped_mm.grouped_mm_bf16() with backend="cake"

the deterministic argument of cake_grouped_mm / cake_grouped_det_mm and the fourth argument of CakeGroupedMm.apply / CakeGroupedDetMm.apply (v0.7.3, v0.7.4rc1: every value ran the deterministic programs)

none: the module is the mode; passing one raises TypeError

flashinfer.grouped_mm.grouped_mm_bf16() with backend="cake" is unchanged: it runs the forward of this module.