flashinfer.cake_dsa_indexer_nondet_train.cake_dsa_indexer_nondet_train_topk¶
- flashinfer.cake_dsa_indexer_nondet_train.cake_dsa_indexer_nondet_train_topk(q: Tensor, k: Tensor, w: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, top_k: int = 2048, softmax_scale: float | None = None, q_causal_offsets: Tensor | None = None, ratio: int = 1, max_seqlen_q: int | None = None, max_seqlen_k: int | None = None, workspace_buffer: Tensor | None = None, indices: Tensor | None = None, scores: Tensor | None = None, return_scores: bool = True) Tuple[Tensor, Tensor | None]¶
Fused DSA indexer scoring and exact top-k selection with set-equality repeatability (DeepSeek Sparse Attention indexer, training), SM100 / SM103 / SM107 only.
For query
tand keyjof the same packed segments[t, j] = sum_h w[t, h] * relu(softmax_scale * sum_d q[t, h, d] * k[j, d])with FP32 products, accumulation and score arithmetic (no BF16 intermediate rounding; the reduction scheme is fixed and documented in
flashinfer/experimental/cake/cake_dsa_indexer_nondet_train/README.md). Every row selects the exact global topmin(top_k, visible)keys of its visible prefix, ranked by score descending then key id descending (+0.0 == -0.0for ranking), and returns them with aligned scores (the computed bits) in no specified slot order. The scores are those offlashinfer.cake_dsa_indexer_det_train(the same scan programs), so a row holds the ids and score bits that package returns for it.This package offers the BF16 form only: float8_e4m3fn
q/kraiseValueError(the FP8 form is an opt-in form of the deterministic package).- Parameters:
q (torch.Tensor) – BF16
[T, 32, 128]indexer queries after RoPE (contiguous).k (torch.Tensor) – BF16
[Tkv, 128]packed indexer keys after normalization and RoPE. A row-strided view (for examplepacked[:, :128]of a[Tkv, 704]tensor) is read in place; the row stride must be a multiple of 8 elements.w (torch.Tensor) – FP32
[T, 32]signed head weights, already scaled by32 ** -0.5.cu_seqlens_q (torch.Tensor) – int32
[S + 1]independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T,cu_seqlens_k[-1] == Tkv). They are not read on the host.cu_seqlens_k (torch.Tensor) – int32
[S + 1]independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T,cu_seqlens_k[-1] == Tkv). They are not read on the host.top_k (int) – Output slots per query,
1 <= top_k <= 4096(default 2048).softmax_scale (Optional[float]) – Positive scale applied to the per-head logits; defaults to
128 ** -0.5.q_causal_offsets (Optional[torch.Tensor]) – int64
[S]per-segment query offsets on the device. Keyjis visible to the query at segment-local positionuiffj < Lkandj < floor((offset + u + 1) / ratio); without offsetsoffset = Lk - Lqforratio == 1(queries are the tail of their key prefix) and0for compressed keys. Negative offsets, empty segments and rows without visible keys are defined (the row is all padding).ratio (int) – Key compression ratio,
>= 1(normally 1).max_seqlen_q (Optional[int]) – Host mirror of the longest query segment; accepted for signature parity and not read by any launch decision.
max_seqlen_k (Optional[int]) – The caller’s bound on every key segment (
max(cu_seqlens_k[1:] - cu_seqlens_k[:-1])); the selected sets are identical with and without it. It lets the dispatch tell a packed call of equal key segments, whose scan then runs without the longest-first segment order; a value aboveTkvor belowceil(Tkv / S)is rejected.workspace_buffer (Optional[torch.Tensor]) – Scratch of at least
cake_dsa_indexer_nondet_train_topk_workspace_size()bytes for the call’s geometry (uint8, 8-byte aligned); allocated through the caching allocator when omitted. A caller-owned buffer may be shared with other calls and kernels between runs: on the ticket architectures the backend then zeroes the buffer’s ticket word before every run, while its own workspace is zeroed once at prepare.indices (Optional[torch.Tensor]) – Caller-owned
[T, top_k]int32 / float32 outputs; allocated when omitted.scores (Optional[torch.Tensor]) – Caller-owned
[T, top_k]int32 / float32 outputs; allocated when omitted.return_scores (bool) –
False: indices only;scoresmust be omitted andNoneis returned in its place (the scores are still computed internally).
- Returns:
indices (torch.Tensor) – int32
[T, top_k]segment-local key ids, unique within the row; themin(top_k, visible)selected ids and-1in the other slots, in no specified slot order.scores (Optional[torch.Tensor]) –
Nonewithreturn_scores=False; else FP32[T, top_k]scores aligned withindices(a selected-0.0stays-0.0); the slots of-1ids hold-inf.
Notes
For identical inputs and configuration (same device, SM count and arguments) the set of
(id, score bits)pairs of every row, and with it the row’s number of valid (non-1) slots, repeats bit for bit; the slot order within a row and the positions of the-1/-infpadding slots may vary between calls. The internal query / key partition and the program the host dispatches for the geometry never change the selected set. Undertorch.use_deterministic_algorithms(True)a call with queries and keys raisesRuntimeErrorbefore it allocates or launches anything (withwarn_only=Trueit warns and runs). The call never materializes a[T, Tkv]matrix, and no call synchronizes with the host, so calls can be captured into a CUDA graph. Finite inputs are the normal domain: with NaN / inf / overflowing inputs the operator returns with the output structure intact and the kernels never hang.