flashinfer.cake_gqa_det_train¶
Cake deterministic GQA attention training for head dimension 256: a varlen
causal forward and backward for SM100a, SM103a and SM107a. The module is a
stable API; its backend package,
flashinfer.experimental.cake.cake_gqa_det_train, is experimental and is
imported on first use.
The mode is part of every name, and no entry takes a deterministic
keyword. Every sum of the forward and the backward has one fixed order:
out, lse, dQ and every dK / dV form (BF16, FP32 window, FP32 row-map
adds with repeated destination rows) repeat bit for bit run to run. The bits
depend only on the shapes, max_seqlen_k and the workspace budget. Under
torch.use_deterministic_algorithms(True), strict or with
warn_only=True, nothing alerts or warns.
The backward’s main kernel sums dK and dV of each key block in tensor memory
and stores dS^T into a workspace, one 32 KiB tile per (head pair, key block
of 128 keys, Q tile of 64 rows); dQ is a GEMM over that workspace. The workspace’s budget_bytes (default 16 GiB) bounds that
storage: a larger call runs in chunks of KV heads, then of key blocks, and
the smallest chunk (one KV head, two key blocks) runs even past it. A
different budget can move the last bits of dQ, not its accuracy. A row-map
destination row becomes ((dst + w_j1) + w_j2) + ... over its window rows
in ascending order. The interface matches the non-deterministic counterpart
flashinfer.cake_gqa_nondet_train.
cake_gqa_det is the autograd entry (its Function is
CakeGQADetFunction); the forward and backward halves, the workspace and
its size are separate entries.
|
Packed-varlen causal GQA attention, head dimension 256, with autograd ( |
|
Forward without autograd, causal with bottom-right alignment per segment. |
|
Backward without autograd: |
|
A workspace ( |
|
Bytes of backward workspace one call needs. |