flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope_varlen¶
- flashinfer.cake_dsa_sparse_attention_nondet_train.cake_dsa_sparse_attention_nondet_nope_varlen(q: Tensor, kv: Tensor, gather_kv_indices: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, max_seqlen_q: int | None = None, max_seqlen_k: int | None = None, *, causal: bool = False, topk_length: Tensor | None = None, softmax_scale: float | None = None, return_lse: bool = False, key_passes: int | None = None, backend: str = 'cake', dkv_acc: Tensor | Callable | None = None, dkv_dst_map: Tensor | None = None, dkv_direct: bool | None = None, check_dst_map: bool | None = None, check_cu_seqlens: bool = False)¶
Packed multi-document form of
cake_dsa_sparse_attention_nondet_nope()(SM100 / SM103 / SM107).Documents are packed along the first dimension of
q(T = cu_seqlens_q[-1]) and ofkv(S = cu_seqlens_k[-1];[S, 512]or[S, 1, 512]);cu_seqlens_qandcu_seqlens_kare independent and query and key lengths may differ per document: the query segment is the tail of its key prefix, so querylocal_qof documentdsits at key position(seqlen_k[d] - seqlen_q[d]) + local_q.gather_kv_indices[T, K_slots]int32 hold key positions relative to the row’s document (-1or>= seqlen_k[d]invalid; anyK_slots >= 1) and are offset bycu_seqlens_kon device. The route follows the table size (cake_backend.NOPE_CONVERT_SLOT_LIMITslots): below the limit, without caller lengths, the forward kernel converts the rows itself in its single launch, position-preserving (the global key or-1at the slot it came from; the row’s length is its last valid slot + 1); at or above it, and always with a callertopk_length, one launch right before the forward kernel compacts each row (valid keys first, in slot order, then-1) so rows with holes cost their valid slots, and the caller’stopk_lengthis a position cut applied before compaction. Zero-length query segments contribute no rows; a row whose every slot is invalid givesout = 0,lse = -infand zero gradients. The kernels apply no positional mask of their own; the offset index rows define the key sets.max_seqlen_q/max_seqlen_kare accepted for signature parity with other varlen attention entry points and are not read on the host. Seecake_dsa_sparse_attention_nondet_nope()for the other arguments, the1 / 16default scale, the FP32dK + dVsemantics and the returns.Under
torch.use_deterministic_algorithms(True)the forward runs and the backward raises PyTorch’sRuntimeErrorbefore its first backward kernel launch (its dK / dV atomics are not run-to-run reproducible); deterministic preparation may run first (see the module docstring). Withwarn_only=Trueit issues aUserWarningand runs. No deterministic package is chosen instead.cu_seqlens_qandcu_seqlens_kare caller-owned prefix sums with these invariants: int32[num_docs + 1]tensors of one length on the device ofq, starting at 0 and non-decreasing,cu_seqlens_q[-1] == T(the packed rows ofq),cu_seqlens_k[-1] == S(the packed rows ofkv; a shorter or longer key window is not accepted) andseqlen_q <= seqlen_kfor every document. Their structure (contiguous int32[num_docs + 1]of one length) is checked on every call; their values only withcheck_cu_seqlens=True, since that check synchronizes with the device, and a violation is otherwise not detected on device: a document withseqlen_k < seqlen_qgives negative own positions (all-invalid rows undercausal), a non-monotonic prefix sum assigns rows to the wrong document. The per-document rows are turned into global key rows on device: by the forward kernel itself below the slot limit (position-preserving, one launch), else compacted in one launch right before the forward kernel; a callertopk_lengthis a position cut applied before compaction and the kernels then read the compacted rows and their counts.- Parameters:
check_cu_seqlens (bool) –
Truechecks the values ofcu_seqlens_q/cu_seqlens_kagainst the invariants above at the cost of one blocking device-to-host copy per call, before any launch (skipped during CUDA-graph capture, where no synchronization is possible; the eager call captured was checked).False(default) keeps the structural checks (dtype, rank, length, contiguity, device) only.causal (bool) – With
False(default) the index row is taken as is – only-1/ out-of-range slots are dropped. WithTruea slot that selects a key after the query’s own position (idx > (seqlen_k[d] - seqlen_q[d]) + local_q) is invalid too, so a top-k selector need not enforce causality itself.dkv_direct (Optional[bool]) – As in
cake_dsa_sparse_attention_nondet_nope():None(default) readsFLASHINFER_CAKE_DSA_DKV_DIRECT;True/Falsedecide explicitly.check_dst_map (Optional[bool]) – As in
cake_dsa_sparse_attention_nondet_nope():None(default) readsFLASHINFER_CAKE_DSA_CHECK_DST_MAP;Truevalidates the values ofdkv_dst_mapon every backward,Falsenever does.