flashinfer.cake_dsa_sparse_attention_det_train.cake_dsa_sparse_attention_det¶
- flashinfer.cake_dsa_sparse_attention_det_train.cake_dsa_sparse_attention_det(q_latent: Tensor, q_rope: Tensor, kv_latent: Tensor, k_rope: Tensor, indices: Tensor, *, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None)¶
Differentiable top-k sparse MLA attention with absorbed queries (DeepSeek Sparse Attention) for training, 64 query heads, SM100 / SM103 / SM107 only; every output repeats bit for bit.
O = softmax(softmax_scale * (q_rope . k_rope^T + q_latent . kv_latent^T)) . kv_latentover the keys a row’sindicesselect (K = V = the latent).Strided inputs are consumed in place (no host copy): the kernels read the query operands through TMA descriptors encoded from the tensor’s own head and token strides and gather the key operands by row stride. Admitted layouts: unit innermost stride; head / token / key-row strides that are positive multiples of 8 elements (16 bytes, the TMA descriptor stride granule); head stride and key row stride at least the slice width; a 16-byte-aligned base address. Outputs equal the contiguous path bitwise.
- Parameters:
q_latent (torch.Tensor) – BF16
[T, 64, 512]absorbed query latents; contiguous or a view of a packed[T, 64, 576]query (head stride 576).q_rope (torch.Tensor) – BF16
[T, 64, 64]query rope part; contiguous, a view of the packed[T, 64, 576]query, or the192:256channel slice of the pre-absorption[T, 64, 256]query (head stride 256, token stride 16384, storage offset 192 elements).kv_latent (torch.Tensor) – BF16
[S, 512]key/value latent (shared K = V); columns0:512of a packed[S, 576]or[S, 704]row (a 128-channel indexer key stored alongside) are accepted.k_rope (torch.Tensor) – BF16
[S, 64]key rope part; columns512:576of the same packed row are accepted.indices (torch.Tensor) – int32
[T, topk]global key rows intokv_latent/k_rope.-1or a value>= Smarks an invalid slot; invalid slots may appear anywhere in the row. Any positivetopk.topk_length (Optional[torch.Tensor]) – int32
[T]; slots>= topk_length[t]are invalid regardless of their content.softmax_scale (Optional[float]) – Defaults to
576 ** -0.5.return_lse (bool) – Also return the natural-log logsumexp
[T, 64]FP32 of the scaled scores over the valid keys (-inffor fully masked rows).key_passes (Optional[int]) – Validated as in the non-deterministic package (
Noneor1 <= key_passes <= S).Nonelets that package’s key-pass policy choose; a count above 1 runs the backward’s main stage as that many key-range passes (a compaction and a main pass each, with their scratch in the backward’s workspace). A scheduling choice only: the gradients are bit for bit those of one pass.backend (str) – Only
"cake".dkv_acc (Optional[torch.Tensor | Callable]) – Caller-owned FP32
[S_dst, >= 576]buffer (row stride a multiple of 4 elements, 16-byte aligned) into which the backward adds the key gradients in place (the caller zeroes it): latent dimensions in columns0:512, rope dimensions in512:576, one FP32 addition per element (an element that receives no contribution keeps its bits). When given, thekv_latent/k_ropegradients of the autograd call areNone. Instead of the tensor, a zero-argument callable that returns it – or an(accumulator, dkv_dst_map)pair – may be given; the backward calls it, so the accumulator need not exist at forward time (under activation checkpointing a tensor passed at forward time is alive for every layer at once). The autograd entries save every tensor they produce for the backward throughctx.save_for_backwardand retain nothing else of a checkpointed forward; a tensordkv_accstays caller-owned.dkv_dst_map (Optional[torch.Tensor]) – int32
[S]destination row ofdkv_accfor every key row (default identity). Repeated destinations are allowed (the rows of a context-parallel window that map onto one parameter row) and combine exactly, as integers, before the one FP32 addition. Every value must lie in[0, S_dst); the kernels do not range-check the map (seecheck_dst_map).dkv_direct (Optional[bool]) – Accepted for signature parity with the non-deterministic package (
None,TrueorFalse) and without effect: the accumulation intodkv_acchas one form here.check_dst_map (Optional[bool]) – Validate the values of
dkv_dst_map(every destination row in[0, S_dst)) on every backward, at the cost of one device synchronization, raisingValueErroron a violation (skipped during CUDA graph capture).None(default) readsFLASHINFER_CAKE_DSA_CHECK_DST_MAP: unset, empty or0(its default) means no check, anything else enables it.True/Falsedecide explicitly and ignore the variable.
- Returns:
out (torch.Tensor) – BF16
[T, 64, 512]; zero for fully masked rows.lse (torch.Tensor) – Only with
return_lse=True.
Notes
Gradients flow to
q_latent,q_rope,kv_latentandk_rope. BF16 operands into the tensor cores, FP32 accumulation; the backward recomputes the scores from the BF16 inputs and forms the exactdeltafrom the saved output residualo_lo. The dK / dV contributions are rounded once to the row’s integer grid and summed as integers; each gradient element is then formed with one more rounding (and, withdkv_acc, one FP32 addition). The backward reuses the storage of the savedo_loas its integer accumulator when it fits (S_dstup to about27.7 Trows) and allocates it otherwise; a second backward through the same graph (retain_graph=True) first recomputes the forward into the saved tensors. The kernels apply no positional mask: the index rows define the key set.Host side: the inputs are validated and bound on the first call for an input geometry (shape, stride, dtype and alignment of every input plus the scale); later calls launch from the remembered plans with freshly allocated outputs (
flashinfer.experimental.cake.cake_dsa_sparse_attention_det_train.cake_backend.BINDING_CACHE, 64 bindings by default, least recently used first out;FLASHINFER_CAKE_DSA_TRAIN_BINDING_CACHE_CAPACITY– a positive integer, read when this module is imported – sets the capacity throughBINDING_CACHE.configure(capacity=...)). No host synchronization unlesscheck_dst_mapasks for one.