flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet

flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet(q_latent: Tensor, q_rope: Tensor, kv_latent: Tensor, k_rope: Tensor, indices: Tensor, *, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None)

Differentiable top-k sparse MLA attention with absorbed queries (DeepSeek Sparse Attention) for training, 64 query heads, SM100 / SM103 / SM107 only.

O = softmax(softmax_scale * (q_rope . k_rope^T + q_latent . kv_latent^T)) . kv_latent over the keys a row’s indices select (K = V = the latent).

Strided inputs are consumed in place (no host copy): the kernels read the query operands through TMA descriptors encoded from the tensor’s own head and token strides and gather the key operands by row stride. Admitted layouts: unit innermost stride; head / token / key-row strides that are positive multiples of 8 elements (16 bytes, the TMA descriptor stride granule); head stride and key row stride at least the slice width; a 16-byte-aligned base address. Outputs equal the contiguous path bitwise (out, lse, dq) or within the FP32 reduction spread (dkv).

Parameters:
  • q_latent (torch.Tensor) – BF16 [T, 64, 512] absorbed query latents; contiguous or a view of a packed [T, 64, 576] query (head stride 576).

  • q_rope (torch.Tensor) – BF16 [T, 64, 64] query rope part; contiguous, a view of the packed [T, 64, 576] query, or the 192:256 channel slice of the pre-absorption [T, 64, 256] query (head stride 256, token stride 16384, storage offset 192 elements).

  • kv_latent (torch.Tensor) – BF16 [S, 512] key/value latent (shared K = V); columns 0:512 of a packed [S, 576] or [S, 704] row (a 128-channel indexer key stored alongside) are accepted.

  • k_rope (torch.Tensor) – BF16 [S, 64] key rope part; columns 512:576 of the same packed row are accepted.

  • indices (torch.Tensor) – int32 [T, topk] global key rows into kv_latent / k_rope. -1 or a value >= S marks an invalid slot; invalid slots may appear anywhere in the row. Any positive topk.

  • topk_length (Optional[torch.Tensor]) – int32 [T]; slots >= topk_length[t] are invalid regardless of their content.

  • softmax_scale (Optional[float]) – Defaults to 576 ** -0.5.

  • return_lse (bool) – Also return the natural-log logsumexp [T, 64] FP32 of the scaled scores over the valid keys (-inf for fully masked rows).

  • key_passes (Optional[int]) – Number of key-range passes of the backward’s main stage. None (default) applies the registered policy: the key range is split into ceil(S * 2304 B / 100 MiB) passes when that exceeds one and the whole row fits the pass workspace budget (T <= 4224 at top-k 2048), so that each pass’s FP32 dK/dV accumulator slice stays L2-resident; otherwise a single pass runs. 1 forces the single pass; a larger value forces that many passes (<= S). The passes carry dQ through an FP32 partial in the workspace (T * 147,456 B plus T * (4 * topk + 4) B, ~608 MiB at T = 4096, top-k 2048); dQ stays bitwise deterministic and the dK/dV accumulation is unchanged.

  • backend (str) – Only "cake".

  • dkv_acc (Optional[torch.Tensor | Callable]) – Caller-owned FP32 [S_dst, >= 576] buffer (row stride a multiple of 4 elements, 16-byte aligned) into which the backward accumulates the key gradients in place (+=; the caller zeroes it): latent dimensions in columns 0:512, rope dimensions in 512:576. When given, the kv_latent / k_rope gradients of the autograd call are None and no BF16 cast runs. Instead of the tensor, a zero-argument callable that returns it – or an (accumulator, dkv_dst_map) pair – may be given; the backward calls it, so the accumulator need not exist at forward time (under activation checkpointing a tensor passed at forward time is alive for every layer at once). The autograd entries save every tensor they produce for the backward through ctx.save_for_backward and retain nothing else of a checkpointed forward; a tensor dkv_acc stays caller-owned.

  • dkv_dst_map (Optional[torch.Tensor]) – int32 [S] destination row of dkv_acc for every key row (default identity). Repeated destinations are allowed (the rows of a context-parallel window that map onto one parameter row) and are combined with FP32 atomics. Every value must lie in [0, S_dst); the kernel does not range-check the map (see check_dst_map).

  • dkv_direct (Optional[bool]) – How the backward accumulates dK / dV into dkv_acc when the registered program has the natural-layout main stages. None (default) reads FLASHINFER_CAKE_DSA_DKV_DIRECT: auto (its default) applies the record’s size rule – the main stage adds into the caller’s packed rows directly (no workspace accumulators, zero fill or cast launch) when the row has at least dkv_direct.min_keys_per_query (4) keys per query token –, 1 forces the direct path whenever the stages are registered, 0 always takes the accumulate-and-cast path; any other value raises ValueError. True / False select the path explicitly and ignore the variable. Both paths add the same FP32 contributions into the same rows (another summation order); without dkv_acc or the natural stages the setting has no effect.

  • check_dst_map (Optional[bool]) – Validate the values of dkv_dst_map (every destination row in [0, S_dst)) on every backward, at the cost of one device synchronization, raising ValueError on a violation. None (default) reads FLASHINFER_CAKE_DSA_CHECK_DST_MAP: unset, empty or 0 (its default) means no check, anything else enables it. True / False decide explicitly and ignore the variable. Leave it off in production.

Returns:

  • out (torch.Tensor) – BF16 [T, 64, 512]; zero for fully masked rows.

  • lse (torch.Tensor) – Only with return_lse=True.

Notes

Gradients flow to q_latent, q_rope, kv_latent and k_rope (dq bitwise deterministic; dkv accumulated in FP32 with atomics, then cast to BF16, or accumulated into dkv_acc). BF16 operands into the tensor cores, FP32 accumulation; the backward recomputes the scores from the BF16 inputs and forms the exact delta from the saved output residual. The kernels apply no positional mask: the index rows define the key set.

Under torch.use_deterministic_algorithms(True) the forward runs and the backward raises PyTorch’s RuntimeError before its first backward kernel launch (its dK / dV atomics are not run-to-run reproducible); deterministic preparation may run first (see the module docstring). With warn_only=True it issues a UserWarning and runs. No deterministic package is chosen instead.

Host side: the inputs are validated and bound on the first call for an input binding (data_ptr, shape, stride and dtype of every input plus the scale); later calls with the same binding launch from the remembered argument plans with freshly allocated outputs (flashinfer.experimental.cake.cake_dsa_sparse_attention_nondet_train.cake_backend.BINDING_CACHE, 64 bindings by default, least recently used first out; FLASHINFER_CAKE_DSA_TRAIN_BINDING_CACHE_CAPACITY – a positive integer, read when this module is imported – sets the capacity through BINDING_CACHE.configure(capacity=...)).