flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope

flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope(q: Tensor, kv: Tensor, indices: Tensor, *, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None)

Differentiable top-k sparse MLA attention without a rope part (NoPE: the whole score is the 512-dimensional latent product) for training, 64 query heads, SM100 / SM103 / SM107 only.

O = softmax(softmax_scale * (q . kv^T)) . kv over the keys a row’s indices select (K = V = the shared latent kv). The native kernels of this form read no rope operands and produce one combined gradient per key row, dkv = dK + dV, natural layout, FP32.

Parameters:
  • q (torch.Tensor) – BF16 [T, 64, 512] absorbed query latents; contiguous or a strided token / head view (unit innermost stride, head and token strides positive multiples of 8 elements, head stride at least 512, 16-byte-aligned base), consumed in place.

  • kv (torch.Tensor) – BF16 [S, 512] shared key/value latent, or its one-head form [S, 1, 512] (squeezed to [S, 512] on the host; the gradient reaches the [S, 1, 512] leaf through that view).

  • indices (torch.Tensor) – int32 [T, K_slots] global key rows into kv. Any K_slots >= 1 (the slot count is a per-row capacity, not a multiple of anything); -1 or a value >= S marks an invalid slot, anywhere in the row. A row whose every slot is invalid gives out = 0, lse = -inf and zero gradients.

  • topk_length (Optional[torch.Tensor]) – int32 [T]; slots >= topk_length[t] are invalid regardless of their content. Without it the route follows the table size (cake_backend.NOPE_CONVERT_SLOT_LIMIT slots): below the limit the per-row lengths (last valid slot + 1) are derived by the forward kernel while it runs and saved for the backward with the rows as given – one launch, a -1 anywhere stays a masked slot; at or above it the rows are compacted on device first (valid keys first in slot order, then -1, the valid count as the length) and the compacted pair is saved, so rows with holes cost their valid slots. Hole-free rows give the same bytes on either route.

  • softmax_scale (Optional[float]) – Defaults to 1 / 16 (512 ** -0.5 of the latent width is not used: the absorbed queries of this form carry their own scale).

  • return_lse (bool) – Also return the natural-log logsumexp [T, 64] FP32 of the scaled scores over the valid keys (-inf for fully masked rows).

  • key_passes (Optional[int]) – Number of key-range passes of the backward’s main stage. None (default) applies the registered policy of this program, which plans one pass on every row (one pass measured fastest on every shape of this form); 1 forces the single pass; a larger value forces that many passes (<= S) over slices of the key range, each pass’s FP32 accumulator slice sized at 2048 B per key. The passes carry dQ through an FP32 partial in the workspace (T * 131,072 B plus T * (4 * K_slots + 4) B, T <= 4736 tokens per chunk at K_slots 2048 / 2176); dQ stays bitwise deterministic and the dK + dV accumulation is unchanged.

  • backend (str) – Only "cake".

  • dkv_acc (Optional[torch.Tensor | Callable]) – Caller-owned FP32 [S_dst, >= 512] buffer (row stride a multiple of 4 elements and at least 512, 16-byte-aligned base) into which the backward accumulates dK + dV in place (+= into columns 0:512; the caller zeroes it). When given, the kv gradient of the autograd call is None. Chunked backward passes over one long row accumulate into one such buffer in FP32 with no intermediate rounding. Instead of the tensor, a zero-argument callable that returns it – or an (accumulator, dkv_dst_map) pair – may be given; the backward calls it, so the accumulator need not exist at forward time (under activation checkpointing a tensor passed at forward time is alive for every layer at once). The autograd entries save every tensor they produce for the backward through ctx.save_for_backward and retain nothing else of a checkpointed forward; a tensor dkv_acc stays caller-owned.

  • dkv_dst_map (Optional[torch.Tensor]) – int32 [S] destination row of dkv_acc for every key row (default identity). Repeated destinations are allowed and combined with FP32 atomics. Every value must lie in [0, S_dst); the kernel does not range-check the map (see check_dst_map).

  • dkv_direct (Optional[bool]) – How the backward accumulates dK + dV into its natural FP32 buffer (dkv_acc or the fresh dkv) when the registered program has the natural-layout main stages. None (default) reads FLASHINFER_CAKE_DSA_DKV_DIRECT: auto (its default) applies the record’s rule – the main stage adds into the buffer directly (no permuted accumulator, no cast launch) when the row has at least dkv_direct.min_keys_per_query keys per query token, 1 for this program (every row with at least as many keys as query tokens; the direct path measured faster than the permuted drain on every such row), so only a problem with more query rows than keys takes the cast –, 1 forces the direct path whenever the stages are registered, 0 always takes the accumulate-and-cast path; any other value raises ValueError. True / False select the path explicitly and ignore the variable. Both paths add the same FP32 contributions into the same rows (another summation order).

  • check_dst_map (Optional[bool]) – Validate the values of dkv_dst_map on every backward (one device synchronization), raising ValueError on a violation. None (default) reads FLASHINFER_CAKE_DSA_CHECK_DST_MAP (unset, empty or 0 = no check); True / False decide explicitly.

Returns:

  • out (torch.Tensor) – BF16 [T, 64, 512]; zero for fully masked rows.

  • lse (torch.Tensor) – Only with return_lse=True.

Notes

Gradients flow to q and kv. dq is BF16 and bitwise deterministic. dkv is accumulated in FP32 by the kernels’ atomics (run-to-run spread within FP32 rounding) and handed over in FP32: through dkv_acc when given, otherwise as the kv gradient – which the autograd engine casts to the dtype of the kv leaf (BF16 for a BF16 leaf); call flashinfer.experimental.cake.cake_dsa_sparse_attention_nondet_train.cake_backend.backward_nope or pass dkv_acc to keep the FP32 values. BF16 operands into the tensor cores, FP32 accumulation and statistics; the backward recomputes the scores from the BF16 inputs and forms the exact delta from the saved output residual. The kernels apply no positional mask: the index rows define the key set.

Under torch.use_deterministic_algorithms(True) the forward runs and the backward raises PyTorch’s RuntimeError before its first backward kernel launch (its dK / dV atomics are not run-to-run reproducible); deterministic preparation may run first (see the module docstring). With warn_only=True it issues a UserWarning and runs. No deterministic package is chosen instead.

The incoming dout may be a head-major view (strides [512, T * 512, 1], the layout an output projection hands back) or any strided [T, 64, 512] view under the layout rules of q (unit channel stride, token and head strides multiples of 8 elements, 16-byte base): it is read in place – the main stage through a stride-carrying tensor map, the delta stage through the token and head strides it receives – with no host copy.

Host side as in cake_dsa_sparse_attention_nondet(): the inputs are validated and bound on the first call for an input binding; later calls with the same binding launch from the remembered argument plans (the same BINDING_CACHE, under keys disjoint from the rope entries).