flashinfer.cake_dsa_sparse_attention_nondet_train

Cake DeepSeek Sparse Attention training (top-k sparse MLA with absorbed queries, 64 query heads): forward and backward for SM100a, SM103a and SM107a. The module is a stable API; its backend package, flashinfer.experimental.cake.cake_dsa_sparse_attention_nondet_train, is experimental and is imported on first use.

The mode is part of every name, and no entry takes a deterministic keyword. The forward and dq are fixed-order. The backward adds the dK / dV contributions into FP32 accumulators with atomics, so those gradients are not bitwise reproducible run to run. Under torch.use_deterministic_algorithms(True) the backward raises PyTorch’s alert before its first backward kernel launch; with warn_only=True it warns and runs. It never falls back to a deterministic implementation. The deterministic package flashinfer.cake_dsa_sparse_attention_det_train has the same entry points (det in place of nondet); its outputs repeat bit for bit run to run.

Each entry is an autograd call. The rope entries score the latent and rope parts; the NoPE entries score the latent part only. The varlen entries take packed documents.

cake_dsa_sparse_attention_nondet(q_latent, ...)

Differentiable top-k sparse MLA attention with absorbed queries (DeepSeek Sparse Attention) for training, 64 query heads, SM100 / SM103 / SM107 only.

cake_dsa_sparse_attention_nondet_varlen(...)

Packed multi-document form of cake_dsa_sparse_attention_nondet() (SM100 / SM103 / SM107).

cake_dsa_sparse_attention_nondet_nope(q, kv, ...)

Differentiable top-k sparse MLA attention without a rope part (NoPE: the whole score is the 512-dimensional latent product) for training, 64 query heads, SM100 / SM103 / SM107 only.

cake_dsa_sparse_attention_nondet_nope_varlen(q, ...)

Packed multi-document form of cake_dsa_sparse_attention_nondet_nope() (SM100 / SM103 / SM107).