flashinfer.cake_gqa_nondet_train¶
Cake non-deterministic GQA attention training for head dimension 256: a varlen
causal forward and backward for SM100a, SM103a and SM107a. The module is a
stable API; its backend package,
flashinfer.experimental.cake.cake_gqa_nondet_train, is experimental and is
imported on first use.
The mode is part of every name, and no entry takes a deterministic
keyword. Every sum of the forward and the backward has one fixed order: the
backward stores dS^T into a workspace chunk sized by the workspace’s
budget_bytes (default 16 GiB), sums dK and dV in tensor memory, computes
dQ as a GEMM over the workspace and folds row-map destinations in window
order. out, lse, dQ and every dK / dV form therefore repeat bit for
bit run to run for one shape, one max_seqlen_k and one budget, and
nothing raises PyTorch’s determinism alert.
cake_gqa_nondet is the autograd entry (its Function is
CakeGQANondetFunction); the forward and backward halves, the workspace and
its size are separate entries.
|
Packed-varlen causal GQA attention, head dimension 256, with autograd ( |
|
Forward without autograd, causal with bottom-right alignment per segment. |
|
Backward without autograd: |
|
A workspace ( |
|
Bytes of device memory one backward call needs beside its operands and outputs. |