flashinfer.cake_gqa_det_train

Cake deterministic GQA attention training for head dimension 256: a varlen causal forward and backward for SM100a, SM103a and SM107a. The module is a stable API; its backend package, flashinfer.experimental.cake.cake_gqa_det_train, is experimental and is imported on first use.

The mode is part of every name, and no entry takes a deterministic keyword. Every sum of the forward and the backward has one fixed order: out, lse, dQ and every dK / dV form (BF16, FP32 window, FP32 row-map adds with repeated destination rows) repeat bit for bit run to run. The bits depend only on the shapes, max_seqlen_k and the workspace budget. Under torch.use_deterministic_algorithms(True), strict or with warn_only=True, nothing alerts or warns.

The backward’s main kernel sums dK and dV of each key block in tensor memory and stores dS^T into a workspace, one 32 KiB tile per (head pair, key block of 128 keys, Q tile of 64 rows); dQ is a GEMM over that workspace. The workspace’s budget_bytes (default 16 GiB) bounds that storage: a larger call runs in chunks of KV heads, then of key blocks, and the smallest chunk (one KV head, two key blocks) runs even past it. A different budget can move the last bits of dQ, not its accuracy. A row-map destination row becomes ((dst + w_j1) + w_j2) + ... over its window rows in ascending order. The interface matches the non-deterministic counterpart flashinfer.cake_gqa_nondet_train.

cake_gqa_det is the autograd entry (its Function is CakeGQADetFunction); the forward and backward halves, the workspace and its size are separate entries.

cake_gqa_det(q, k, v, cu_seqlens_q, ...[, ...])

Packed-varlen causal GQA attention, head dimension 256, with autograd (CakeGQADetFunction).

cake_gqa_det_train_forward(q, k, v, ...[, ...])

Forward without autograd, causal with bottom-right alignment per segment.

cake_gqa_det_train_backward(q, k, v, out, ...)

Backward without autograd: (dq, dk, dv) from the forward's out and lse.

cake_gqa_det_train_workspace([device, ...])

A workspace (CakeGQADetWorkspace) for cake_gqa_det() and its two halves.

cake_gqa_det_train_workspace_bytes(total_q, ...)

Bytes of backward workspace one call needs.