flashinfer.cake_dsa_indexer_nondet_train.cake_dsa_indexer_nondet_train_topk

flashinfer.cake_dsa_indexer_nondet_train.cake_dsa_indexer_nondet_train_topk(q: Tensor, k: Tensor, w: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, top_k: int = 2048, softmax_scale: float | None = None, q_causal_offsets: Tensor | None = None, ratio: int = 1, max_seqlen_q: int | None = None, max_seqlen_k: int | None = None, workspace_buffer: Tensor | None = None, indices: Tensor | None = None, scores: Tensor | None = None, return_scores: bool = True) → Tuple[Tensor, Tensor | None]

Fused DSA indexer scoring and exact top-k selection with set-equality repeatability (DeepSeek Sparse Attention indexer, training), SM100 / SM103 / SM107 only.

For query t and key j of the same packed segment

s[t, j] = sum_h w[t, h] * relu(softmax_scale * sum_d q[t, h, d] * k[j, d])

with FP32 products, accumulation and score arithmetic (no BF16 intermediate rounding; the reduction scheme is fixed and documented in flashinfer/experimental/cake/cake_dsa_indexer_nondet_train/README.md). Every row selects the exact global top min(top_k, visible) keys of its visible prefix, ranked by score descending then key id descending (+0.0 == -0.0 for ranking), and returns them with aligned scores (the computed bits) in no specified slot order. The scores are those of flashinfer.cake_dsa_indexer_det_train (the same scan programs), so a row holds the ids and score bits that package returns for it.

This package offers the BF16 form only: float8_e4m3fn q / k raise ValueError (the FP8 form is an opt-in form of the deterministic package).

Parameters:
  • q (torch.Tensor) – BF16 [T, 32, 128] indexer queries after RoPE (contiguous).

  • k (torch.Tensor) – BF16 [Tkv, 128] packed indexer keys after normalization and RoPE. A row-strided view (for example packed[:, :128] of a [Tkv, 704] tensor) is read in place; the row stride must be a multiple of 8 elements.

  • w (torch.Tensor) – FP32 [T, 32] signed head weights, already scaled by 32 ** -0.5.

  • cu_seqlens_q (torch.Tensor) – int32 [S + 1] independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T, cu_seqlens_k[-1] == Tkv). They are not read on the host.

  • cu_seqlens_k (torch.Tensor) – int32 [S + 1] independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T, cu_seqlens_k[-1] == Tkv). They are not read on the host.

  • top_k (int) – Output slots per query, 1 <= top_k <= 4096 (default 2048).

  • softmax_scale (Optional[float]) – Positive scale applied to the per-head logits; defaults to 128 ** -0.5.

  • q_causal_offsets (Optional[torch.Tensor]) – int64 [S] per-segment query offsets on the device. Key j is visible to the query at segment-local position u iff j < Lk and j < floor((offset + u + 1) / ratio); without offsets offset = Lk - Lq for ratio == 1 (queries are the tail of their key prefix) and 0 for compressed keys. Negative offsets, empty segments and rows without visible keys are defined (the row is all padding).

  • ratio (int) – Key compression ratio, >= 1 (normally 1).

  • max_seqlen_q (Optional[int]) – Host mirror of the longest query segment; accepted for signature parity and not read by any launch decision.

  • max_seqlen_k (Optional[int]) – The caller’s bound on every key segment (max(cu_seqlens_k[1:] - cu_seqlens_k[:-1])); the selected sets are identical with and without it. It lets the dispatch tell a packed call of equal key segments, whose scan then runs without the longest-first segment order; a value above Tkv or below ceil(Tkv / S) is rejected.

  • workspace_buffer (Optional[torch.Tensor]) – Scratch of at least cake_dsa_indexer_nondet_train_topk_workspace_size() bytes for the call’s geometry (uint8, 8-byte aligned); allocated through the caching allocator when omitted. A caller-owned buffer may be shared with other calls and kernels between runs: on the ticket architectures the backend then zeroes the buffer’s ticket word before every run, while its own workspace is zeroed once at prepare.

  • indices (Optional[torch.Tensor]) – Caller-owned [T, top_k] int32 / float32 outputs; allocated when omitted.

  • scores (Optional[torch.Tensor]) – Caller-owned [T, top_k] int32 / float32 outputs; allocated when omitted.

  • return_scores (bool) – False: indices only; scores must be omitted and None is returned in its place (the scores are still computed internally).

Returns:

  • indices (torch.Tensor) – int32 [T, top_k] segment-local key ids, unique within the row; the min(top_k, visible) selected ids and -1 in the other slots, in no specified slot order.

  • scores (Optional[torch.Tensor]) – None with return_scores=False; else FP32 [T, top_k] scores aligned with indices (a selected -0.0 stays -0.0); the slots of -1 ids hold -inf.

Notes

For identical inputs and configuration (same device, SM count and arguments) the set of (id, score bits) pairs of every row, and with it the row’s number of valid (non -1) slots, repeats bit for bit; the slot order within a row and the positions of the -1 / -inf padding slots may vary between calls. The internal query / key partition and the program the host dispatches for the geometry never change the selected set. Under torch.use_deterministic_algorithms(True) a call with queries and keys raises RuntimeError before it allocates or launches anything (with warn_only=True it warns and runs). The call never materializes a [T, Tkv] matrix, and no call synchronizes with the host, so calls can be captured into a CUDA graph. Finite inputs are the normal domain: with NaN / inf / overflowing inputs the operator returns with the output structure intact and the kernels never hang.