flashinfer.cake_dsa_indexer_det_train.cake_dsa_indexer_det_train_topk¶
- flashinfer.cake_dsa_indexer_det_train.cake_dsa_indexer_det_train_topk(q: Tensor, k: Tensor, w: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, top_k: int = 2048, softmax_scale: float | None = None, q_causal_offsets: Tensor | None = None, ratio: int = 1, max_seqlen_q: int | None = None, max_seqlen_k: int | None = None, workspace_buffer: Tensor | None = None, indices: Tensor | None = None, scores: Tensor | None = None, return_scores: bool = True, k_scale: Tensor | None = None) Tuple[Tensor, Tensor | None]¶
Fused DSA indexer scoring and deterministic exact top-k selection (DeepSeek Sparse Attention indexer, training), SM100 / SM103 / SM107 only.
For query
tand keyjof the same packed segments[t, j] = sum_h w[t, h] * relu(softmax_scale * sum_d q[t, h, d] * k[j, d])with FP32 products, accumulation and score arithmetic (no BF16 intermediate rounding; the reduction scheme is fixed and documented in
flashinfer/experimental/cake/cake_dsa_indexer_det_train/README.md). Every row selects the exact global topmin(top_k, visible)keys of its visible prefix, ranked by score descending then key id descending (+0.0 == -0.0for ranking), and returns them in ascending id order with aligned scores.The FP8 form is selected by float8_e4m3fn
qandkwith a per-key FP32k_scale:s[t, j] = k_scale[j] * sum_h w[t, h] * relu(sum_d q[t, h, d] * k[j, d])where
wcarries every other scale (softmax_scalemust beNone). The e4m3 products are exact and accumulate in FP32 on the tensor cores: four K = 32 steps on SM100 / SM103, in the order of DeepGEMM’sfp8_mqa_logits(bit-identical logits), and two K = 64 steps on SM107, whose logits can differ from DeepGEMM’s in the last bits. Every FP8 program of one architecture gives the same ids and score bits; selection, visibility and outputs follow the BF16 form.- Parameters:
q (torch.Tensor) – BF16
[T, 32, 128]indexer queries after RoPE (contiguous); float8_e4m3fn for the FP8 form.k (torch.Tensor) – BF16
[Tkv, 128]packed indexer keys after normalization and RoPE. A row-strided view (for examplepacked[:, :128]of a[Tkv, 704]tensor) is read in place; the row stride must be a multiple of 8 elements. The FP8 form: float8_e4m3fn (the same dtype asq), a row stride that is a multiple of 16 elements.w (torch.Tensor) – FP32
[T, 32]signed head weights, already scaled by32 ** -0.5.cu_seqlens_q (torch.Tensor) – int32
[S + 1]independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T,cu_seqlens_k[-1] == Tkv). They are not read on the host.cu_seqlens_k (torch.Tensor) – int32
[S + 1]independent query and key segment boundaries on the device (cu_seqlens_q[-1] == T,cu_seqlens_k[-1] == Tkv). They are not read on the host.top_k (int) – Output slots per query,
1 <= top_k <= 4096(default 2048).softmax_scale (Optional[float]) – Positive scale applied to the per-head logits; defaults to
128 ** -0.5. The FP8 form takesNone(the scale is folded intow).q_causal_offsets (Optional[torch.Tensor]) – int64
[S]per-segment query offsets on the device. Keyjis visible to the query at segment-local positionuiffj < Lkandj < floor((offset + u + 1) / ratio); without offsetsoffset = Lk - Lqforratio == 1(queries are the tail of their key prefix) and0for compressed keys. Negative offsets, empty segments and rows without visible keys are defined (the row is all padding).ratio (int) – Key compression ratio,
>= 1(normally 1).max_seqlen_q (Optional[int]) – Host mirror of the longest query segment; accepted for signature parity and not read by any launch decision.
max_seqlen_k (Optional[int]) – The caller’s bound on every key segment (
max(cu_seqlens_k[1:] - cu_seqlens_k[:-1])); the results are bitwise identical with and without it. It sizes the rank finalize’s bitmap pool and so selects the finalize program variant. Onsm_100aandsm_103ait also shows the host whether every key segment of a packed call has the same length (Tkv == S * max_seqlen_k): such a call runs its scan program without the longest-segment-first unit order (the:lsfcopy). A value aboveTkvor belowceil(Tkv / S)is rejected.workspace_buffer (Optional[torch.Tensor]) – Scratch of at least
cake_dsa_indexer_det_train_topk_workspace_size()bytes for the call’s geometry (uint8, 8-byte aligned); allocated through the caching allocator when omitted. A caller-owned buffer may be shared with other calls and kernels between runs: on the ticket architectures the backend then zeroes the buffer’s ticket word before every run, while its own workspace is zeroed once at prepare.indices (Optional[torch.Tensor]) – Caller-owned
[T, top_k]int32 output; allocated when omitted.scores (Optional[torch.Tensor]) – Caller-owned
[T, top_k]float32 output; allocated when omitted.return_scores (bool) –
False: indices only;scoresmust be omitted andNoneis returned in its place (the scores are still computed internally).k_scale (Optional[torch.Tensor]) – FP32
[Tkv]per-key dequantization scale of the FP8 form (contiguous). BF16 keys take none.
- Returns:
indices (torch.Tensor) – int32
[T, top_k]segment-local key ids, ascending within the row, unique; unused tail slots hold-1.scores (Optional[torch.Tensor]) –
Nonewithreturn_scores=False; else FP32[T, top_k]scores aligned withindices(a selected-0.0stays-0.0); tail slots hold-inf.
Notes
Identical inputs give identical ids and score bits; the result does not depend on the internal query / key partition or on the program the host dispatches for the geometry. The call never materializes a
[T, Tkv]matrix, and no call synchronizes with the host, so calls can be captured into a CUDA graph. Finite inputs are the normal domain: with NaN / inf / overflowing inputs the operator returns with the output structure intact and the kernels never hang.