flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope¶
- flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope(q: Tensor, kv: Tensor, indices: Tensor, *, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None)¶
Differentiable top-k sparse MLA attention without a rope part (NoPE: the whole score is the 512-dimensional latent product) for training, 64 query heads, SM100 / SM103 / SM107 only.
O = softmax(softmax_scale * (q . kv^T)) . kvover the keys a row’sindicesselect (K = V = the shared latentkv). The native kernels of this form read no rope operands and produce one combined gradient per key row,dkv = dK + dV, natural layout, FP32.- Parameters:
q (torch.Tensor) – BF16
[T, 64, 512]absorbed query latents; contiguous or a strided token / head view (unit innermost stride, head and token strides positive multiples of 8 elements, head stride at least 512, 16-byte-aligned base), consumed in place.kv (torch.Tensor) – BF16
[S, 512]shared key/value latent, or its one-head form[S, 1, 512](squeezed to[S, 512]on the host; the gradient reaches the[S, 1, 512]leaf through that view).indices (torch.Tensor) – int32
[T, K_slots]global key rows intokv. AnyK_slots >= 1(the slot count is a per-row capacity, not a multiple of anything);-1or a value>= Smarks an invalid slot, anywhere in the row. A row whose every slot is invalid givesout = 0,lse = -infand zero gradients.topk_length (Optional[torch.Tensor]) – int32
[T]; slots>= topk_length[t]are invalid regardless of their content. Without it the route follows the table size (cake_backend.NOPE_CONVERT_SLOT_LIMITslots): below the limit the per-row lengths (last valid slot + 1) are derived by the forward kernel while it runs and saved for the backward with the rows as given – one launch, a-1anywhere stays a masked slot; at or above it the rows are compacted on device first (valid keys first in slot order, then-1, the valid count as the length) and the compacted pair is saved, so rows with holes cost their valid slots. Hole-free rows give the same bytes on either route.softmax_scale (Optional[float]) – Defaults to
1 / 16(512 ** -0.5of the latent width is not used: the absorbed queries of this form carry their own scale).return_lse (bool) – Also return the natural-log logsumexp
[T, 64]FP32 of the scaled scores over the valid keys (-inffor fully masked rows).key_passes (Optional[int]) – Number of key-range passes of the backward’s main stage.
None(default) applies the registered policy of this program, which plans one pass on every row (one pass measured fastest on every shape of this form);1forces the single pass; a larger value forces that many passes (<= S) over slices of the key range, each pass’s FP32 accumulator slice sized at 2048 B per key. The passes carry dQ through an FP32 partial in the workspace (T * 131,072B plusT * (4 * K_slots + 4)B,T <= 4736tokens per chunk atK_slots2048 / 2176); dQ stays bitwise deterministic and the dK + dV accumulation is unchanged.backend (str) – Only
"cake".dkv_acc (Optional[torch.Tensor | Callable]) – Caller-owned FP32
[S_dst, >= 512]buffer (row stride a multiple of 4 elements and at least 512, 16-byte-aligned base) into which the backward accumulatesdK + dVin place (+=into columns0:512; the caller zeroes it). When given, thekvgradient of the autograd call isNone. Chunked backward passes over one long row accumulate into one such buffer in FP32 with no intermediate rounding. Instead of the tensor, a zero-argument callable that returns it – or an(accumulator, dkv_dst_map)pair – may be given; the backward calls it, so the accumulator need not exist at forward time (under activation checkpointing a tensor passed at forward time is alive for every layer at once). The autograd entries save every tensor they produce for the backward throughctx.save_for_backwardand retain nothing else of a checkpointed forward; a tensordkv_accstays caller-owned.dkv_dst_map (Optional[torch.Tensor]) – int32
[S]destination row ofdkv_accfor every key row (default identity). Repeated destinations are allowed and combined with FP32 atomics. Every value must lie in[0, S_dst); the kernel does not range-check the map (seecheck_dst_map).dkv_direct (Optional[bool]) – How the backward accumulates
dK + dVinto its natural FP32 buffer (dkv_accor the freshdkv) when the registered program has the natural-layout main stages.None(default) readsFLASHINFER_CAKE_DSA_DKV_DIRECT:auto(its default) applies the record’s rule – the main stage adds into the buffer directly (no permuted accumulator, no cast launch) when the row has at leastdkv_direct.min_keys_per_querykeys per query token, 1 for this program (every row with at least as many keys as query tokens; the direct path measured faster than the permuted drain on every such row), so only a problem with more query rows than keys takes the cast –,1forces the direct path whenever the stages are registered,0always takes the accumulate-and-cast path; any other value raisesValueError.True/Falseselect the path explicitly and ignore the variable. Both paths add the same FP32 contributions into the same rows (another summation order).check_dst_map (Optional[bool]) – Validate the values of
dkv_dst_mapon every backward (one device synchronization), raisingValueErroron a violation.None(default) readsFLASHINFER_CAKE_DSA_CHECK_DST_MAP(unset, empty or0= no check);True/Falsedecide explicitly.
- Returns:
out (torch.Tensor) – BF16
[T, 64, 512]; zero for fully masked rows.lse (torch.Tensor) – Only with
return_lse=True.
Notes
Gradients flow to
qandkv.dqis BF16 and bitwise deterministic.dkvis accumulated in FP32 by the kernels’ atomics (run-to-run spread within FP32 rounding) and handed over in FP32: throughdkv_accwhen given, otherwise as thekvgradient – which the autograd engine casts to the dtype of thekvleaf (BF16 for a BF16 leaf); callflashinfer.experimental.cake.cake_dsa_sparse_attention_nondet_train.cake_backend.backward_nopeor passdkv_accto keep the FP32 values. BF16 operands into the tensor cores, FP32 accumulation and statistics; the backward recomputes the scores from the BF16 inputs and forms the exactdeltafrom the saved output residual. The kernels apply no positional mask: the index rows define the key set.Under
torch.use_deterministic_algorithms(True)the forward runs and the backward raises PyTorch’sRuntimeErrorbefore its first backward kernel launch (its dK / dV atomics are not run-to-run reproducible); deterministic preparation may run first (see the module docstring). Withwarn_only=Trueit issues aUserWarningand runs. No deterministic package is chosen instead.The incoming
doutmay be a head-major view (strides[512, T * 512, 1], the layout an output projection hands back) or any strided[T, 64, 512]view under the layout rules ofq(unit channel stride, token and head strides multiples of 8 elements, 16-byte base): it is read in place – the main stage through a stride-carrying tensor map, the delta stage through the token and head strides it receives – with no host copy.Host side as in
cake_dsa_sparse_attention_nondet(): the inputs are validated and bound on the first call for an input binding; later calls with the same binding launch from the remembered argument plans (the sameBINDING_CACHE, under keys disjoint from the rope entries).