flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet¶
- flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet(q_latent: Tensor, q_rope: Tensor, kv_latent: Tensor, k_rope: Tensor, indices: Tensor, *, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None)¶
Differentiable top-k sparse MLA attention with absorbed queries (DeepSeek Sparse Attention) for training, 64 query heads, SM100 / SM103 / SM107 only.
O = softmax(softmax_scale * (q_rope . k_rope^T + q_latent . kv_latent^T)) . kv_latentover the keys a row’sindicesselect (K = V = the latent).Strided inputs are consumed in place (no host copy): the kernels read the query operands through TMA descriptors encoded from the tensor’s own head and token strides and gather the key operands by row stride. Admitted layouts: unit innermost stride; head / token / key-row strides that are positive multiples of 8 elements (16 bytes, the TMA descriptor stride granule); head stride and key row stride at least the slice width; a 16-byte-aligned base address. Outputs equal the contiguous path bitwise (
out,lse,dq) or within the FP32 reduction spread (dkv).- Parameters:
q_latent (torch.Tensor) – BF16
[T, 64, 512]absorbed query latents; contiguous or a view of a packed[T, 64, 576]query (head stride 576).q_rope (torch.Tensor) – BF16
[T, 64, 64]query rope part; contiguous, a view of the packed[T, 64, 576]query, or the192:256channel slice of the pre-absorption[T, 64, 256]query (head stride 256, token stride 16384, storage offset 192 elements).kv_latent (torch.Tensor) – BF16
[S, 512]key/value latent (shared K = V); columns0:512of a packed[S, 576]or[S, 704]row (a 128-channel indexer key stored alongside) are accepted.k_rope (torch.Tensor) – BF16
[S, 64]key rope part; columns512:576of the same packed row are accepted.indices (torch.Tensor) – int32
[T, topk]global key rows intokv_latent/k_rope.-1or a value>= Smarks an invalid slot; invalid slots may appear anywhere in the row. Any positivetopk.topk_length (Optional[torch.Tensor]) – int32
[T]; slots>= topk_length[t]are invalid regardless of their content.softmax_scale (Optional[float]) – Defaults to
576 ** -0.5.return_lse (bool) – Also return the natural-log logsumexp
[T, 64]FP32 of the scaled scores over the valid keys (-inffor fully masked rows).key_passes (Optional[int]) – Number of key-range passes of the backward’s main stage.
None(default) applies the registered policy: the key range is split intoceil(S * 2304 B / 100 MiB)passes when that exceeds one and the whole row fits the pass workspace budget (T <= 4224at top-k 2048), so that each pass’s FP32 dK/dV accumulator slice stays L2-resident; otherwise a single pass runs.1forces the single pass; a larger value forces that many passes (<= S). The passes carry dQ through an FP32 partial in the workspace (T * 147,456B plusT * (4 * topk + 4)B, ~608 MiB atT = 4096, top-k 2048); dQ stays bitwise deterministic and the dK/dV accumulation is unchanged.backend (str) – Only
"cake".dkv_acc (Optional[torch.Tensor | Callable]) – Caller-owned FP32
[S_dst, >= 576]buffer (row stride a multiple of 4 elements, 16-byte aligned) into which the backward accumulates the key gradients in place (+=; the caller zeroes it): latent dimensions in columns0:512, rope dimensions in512:576. When given, thekv_latent/k_ropegradients of the autograd call areNoneand no BF16 cast runs. Instead of the tensor, a zero-argument callable that returns it – or an(accumulator, dkv_dst_map)pair – may be given; the backward calls it, so the accumulator need not exist at forward time (under activation checkpointing a tensor passed at forward time is alive for every layer at once). The autograd entries save every tensor they produce for the backward throughctx.save_for_backwardand retain nothing else of a checkpointed forward; a tensordkv_accstays caller-owned.dkv_dst_map (Optional[torch.Tensor]) – int32
[S]destination row ofdkv_accfor every key row (default identity). Repeated destinations are allowed (the rows of a context-parallel window that map onto one parameter row) and are combined with FP32 atomics. Every value must lie in[0, S_dst); the kernel does not range-check the map (seecheck_dst_map).dkv_direct (Optional[bool]) – How the backward accumulates dK / dV into
dkv_accwhen the registered program has the natural-layout main stages.None(default) readsFLASHINFER_CAKE_DSA_DKV_DIRECT:auto(its default) applies the record’s size rule – the main stage adds into the caller’s packed rows directly (no workspace accumulators, zero fill or cast launch) when the row has at leastdkv_direct.min_keys_per_query(4) keys per query token –,1forces the direct path whenever the stages are registered,0always takes the accumulate-and-cast path; any other value raisesValueError.True/Falseselect the path explicitly and ignore the variable. Both paths add the same FP32 contributions into the same rows (another summation order); withoutdkv_accor the natural stages the setting has no effect.check_dst_map (Optional[bool]) – Validate the values of
dkv_dst_map(every destination row in[0, S_dst)) on every backward, at the cost of one device synchronization, raisingValueErroron a violation.None(default) readsFLASHINFER_CAKE_DSA_CHECK_DST_MAP: unset, empty or0(its default) means no check, anything else enables it.True/Falsedecide explicitly and ignore the variable. Leave it off in production.
- Returns:
out (torch.Tensor) – BF16
[T, 64, 512]; zero for fully masked rows.lse (torch.Tensor) – Only with
return_lse=True.
Notes
Gradients flow to
q_latent,q_rope,kv_latentandk_rope(dqbitwise deterministic;dkvaccumulated in FP32 with atomics, then cast to BF16, or accumulated intodkv_acc). BF16 operands into the tensor cores, FP32 accumulation; the backward recomputes the scores from the BF16 inputs and forms the exactdeltafrom the saved output residual. The kernels apply no positional mask: the index rows define the key set.Under
torch.use_deterministic_algorithms(True)the forward runs and the backward raises PyTorch’sRuntimeErrorbefore its first backward kernel launch (its dK / dV atomics are not run-to-run reproducible); deterministic preparation may run first (see the module docstring). Withwarn_only=Trueit issues aUserWarningand runs. No deterministic package is chosen instead.Host side: the inputs are validated and bound on the first call for an input binding (
data_ptr, shape, stride and dtype of every input plus the scale); later calls with the same binding launch from the remembered argument plans with freshly allocated outputs (flashinfer.experimental.cake.cake_dsa_sparse_attention_nondet_train.cake_backend.BINDING_CACHE, 64 bindings by default, least recently used first out;FLASHINFER_CAKE_DSA_TRAIN_BINDING_CACHE_CAPACITY– a positive integer, read when this module is imported – sets the capacity throughBINDING_CACHE.configure(capacity=...)).