flashinfer.cake_dsa_indexer_det_train.cake_dsa_indexer_det_train_topk

flashinfer.cake_dsa_indexer_det_train.cake_dsa_indexer_det_train_topk(q: Tensor, k: Tensor, w: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, top_k: int = 2048, softmax_scale: float | None = None, q_causal_offsets: Tensor | None = None, ratio: int = 1, max_seqlen_q: int | None = None, max_seqlen_k: int | None = None, workspace_buffer: Tensor | None = None, indices: Tensor | None = None, scores: Tensor | None = None, return_scores: bool = True, k_scale: Tensor | None = None) → Tuple[Tensor, Tensor | None]

Fused DSA indexer scoring and deterministic exact top-k selection (DeepSeek Sparse Attention indexer, training), SM100 / SM103 / SM107 only.

For query t and key j of the same packed segment

s[t, j] = sum_h w[t, h] * relu(softmax_scale * sum_d q[t, h, d] * k[j, d])

with FP32 products, accumulation and score arithmetic (no BF16 intermediate rounding; the reduction scheme is fixed and documented in flashinfer/experimental/cake/cake_dsa_indexer_det_train/README.md). Every row selects the exact global top min(top_k, visible) keys of its visible prefix, ranked by score descending then key id descending (+0.0 == -0.0 for ranking), and returns them in ascending id order with aligned scores.

The FP8 form is selected by float8_e4m3fn q and k with a per-key FP32 k_scale:

s[t, j] = k_scale[j] * sum_h w[t, h] * relu(sum_d q[t, h, d] * k[j, d])

where w carries every other scale (softmax_scale must be None). The e4m3 products are exact and accumulate in FP32 on the tensor cores: four K = 32 steps on SM100 / SM103, in the order of DeepGEMM’s fp8_mqa_logits (bit-identical logits), and two K = 64 steps on SM107, whose logits can differ from DeepGEMM’s in the last bits. Every FP8 program of one architecture gives the same ids and score bits; selection, visibility and outputs follow the BF16 form.

Parameters:
  • q (torch.Tensor) – BF16 [T, 32, 128] indexer queries after RoPE (contiguous); float8_e4m3fn for the FP8 form.

  • k (torch.Tensor) – BF16 [Tkv, 128] packed indexer keys after normalization and RoPE. A row-strided view (for example packed[:, :128] of a [Tkv, 704] tensor) is read in place; the row stride must be a multiple of 8 elements. The FP8 form: float8_e4m3fn (the same dtype as q), a row stride that is a multiple of 16 elements.

  • w (torch.Tensor) – FP32 [T, 32] signed head weights, already scaled by 32 ** -0.5.

  • cu_seqlens_q (torch.Tensor) – int32 [S + 1] independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T, cu_seqlens_k[-1] == Tkv). They are not read on the host.

  • cu_seqlens_k (torch.Tensor) – int32 [S + 1] independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T, cu_seqlens_k[-1] == Tkv). They are not read on the host.

  • top_k (int) – Output slots per query, 1 <= top_k <= 4096 (default 2048).

  • softmax_scale (Optional[float]) – Positive scale applied to the per-head logits; defaults to 128 ** -0.5. The FP8 form takes None (the scale is folded into w).

  • q_causal_offsets (Optional[torch.Tensor]) – int64 [S] per-segment query offsets on the device. Key j is visible to the query at segment-local position u iff j < Lk and j < floor((offset + u + 1) / ratio); without offsets offset = Lk - Lq for ratio == 1 (queries are the tail of their key prefix) and 0 for compressed keys. Negative offsets, empty segments and rows without visible keys are defined (the row is all padding).

  • ratio (int) – Key compression ratio, >= 1 (normally 1).

  • max_seqlen_q (Optional[int]) – Host mirror of the longest query segment; accepted for signature parity and not read by any launch decision.

  • max_seqlen_k (Optional[int]) – The caller’s bound on every key segment (max(cu_seqlens_k[1:] - cu_seqlens_k[:-1])); the results are bitwise identical with and without it. It sizes the rank finalize’s bitmap pool and so selects the finalize program variant. On sm_100a and sm_103a it also shows the host whether every key segment of a packed call has the same length (Tkv == S * max_seqlen_k): such a call runs its scan program without the longest-segment-first unit order (the :lsf copy). A value above Tkv or below ceil(Tkv / S) is rejected.

  • workspace_buffer (Optional[torch.Tensor]) – Scratch of at least cake_dsa_indexer_det_train_topk_workspace_size() bytes for the call’s geometry (uint8, 8-byte aligned); allocated through the caching allocator when omitted. A caller-owned buffer may be shared with other calls and kernels between runs: on the ticket architectures the backend then zeroes the buffer’s ticket word before every run, while its own workspace is zeroed once at prepare.

  • indices (Optional[torch.Tensor]) – Caller-owned [T, top_k] int32 output; allocated when omitted.

  • scores (Optional[torch.Tensor]) – Caller-owned [T, top_k] float32 output; allocated when omitted.

  • return_scores (bool) – False: indices only; scores must be omitted and None is returned in its place (the scores are still computed internally).

  • k_scale (Optional[torch.Tensor]) – FP32 [Tkv] per-key dequantization scale of the FP8 form (contiguous). BF16 keys take none.

Returns:

  • indices (torch.Tensor) – int32 [T, top_k] segment-local key ids, ascending within the row, unique; unused tail slots hold -1.

  • scores (Optional[torch.Tensor]) – None with return_scores=False; else FP32 [T, top_k] scores aligned with indices (a selected -0.0 stays -0.0); tail slots hold -inf.

Notes

Identical inputs give identical ids and score bits; the result does not depend on the internal query / key partition or on the program the host dispatches for the geometry. The call never materializes a [T, Tkv] matrix, and no call synchronizes with the host, so calls can be captured into a CUDA graph. Finite inputs are the normal domain: with NaN / inf / overflowing inputs the operator returns with the output structure intact and the kernels never hang.