flashinfer.cake_dsa_sparse_attention_det_train¶
Cake DeepSeek Sparse Attention training (top-k sparse MLA with absorbed
queries, 64 query heads): forward and backward for SM100a, SM103a and SM107a.
The module is a stable API; its backend package,
flashinfer.experimental.cake.cake_dsa_sparse_attention_det_train, is
experimental and is imported on first use.
The mode is part of every name, and no entry takes a deterministic
keyword. Every output repeats bit for bit for the same inputs, device
architecture and package version: the forward and dq are fixed-order, and
the backward adds the dK / dV contributions as integers on a power-of-two grid
per destination row, so the order of the additions does not change the
result. Nothing raises or warns under
torch.use_deterministic_algorithms(True). The entries take the same
arguments as those of flashinfer.cake_dsa_sparse_attention_nondet_train.
Each entry is an autograd call. The rope entries score the latent and rope parts; the NoPE entries score the latent part only. The varlen entries take packed documents.
|
Differentiable top-k sparse MLA attention with absorbed queries (DeepSeek Sparse Attention) for training, 64 query heads, SM100 / SM103 / SM107 only; every output repeats bit for bit. |
|
Packed multi-document form of |
|
Differentiable top-k sparse MLA attention without a rope part (NoPE: the whole score is the 512-dimensional latent product) for training, 64 query heads, SM100 / SM103 / SM107 only; every output repeats bit for bit. |
Packed multi-document form of |