flashinfer.cake_gqa_det_train.cake_gqa_det¶
- flashinfer.cake_gqa_det_train.cake_gqa_det(q: Tensor, k: Tensor, v: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, softmax_scale: float | None = None, causal: bool = True, dkv_acc: Callable[[], tuple] | None = None, max_seqlen_k: int | None = None, workspace: Any | None = None, return_lse: bool = False, cu_seqlens_q_host: Sequence[int] | Tensor | None = None, cu_seqlens_k_host: Sequence[int] | Tensor | None = None) Tensor | tuple[Tensor, Tensor]¶
Packed-varlen causal GQA attention, head dimension 256, with autograd (
CakeGQADetFunction).- Parameters:
q (torch.Tensor) – BF16
[T_q, H_q, 256]queries.k (torch.Tensor) – BF16
[T_k, H_kv, 256]keys and values; query headhreads key headh / (H_q / H_kv).v (torch.Tensor) – BF16
[T_k, H_kv, 256]keys and values; query headhreads key headh / (H_q / H_kv).cu_seqlens_q (torch.Tensor) – int32
[B + 1]on the device,cu[0] = 0, non-decreasing,cu_q[B] = T_q,cu_k[B] = T_k.cu_seqlens_k (torch.Tensor) – int32
[B + 1]on the device,cu[0] = 0, non-decreasing,cu_q[B] = T_q,cu_k[B] = T_k.softmax_scale (float, optional) – Default
256 ** -0.5.causal (bool) – Must be
True(bottom-right alignment per segment).dkv_acc (callable, optional) –
None:k/vget BF16 gradients (FP32 sums over the query heads rounded once). A zero-argument callable (CakeGQADetDkvAccSource), called in backward, returning(dk_acc, dv_acc, row_map)adds the FP32 window gradients into the caller’s FP32[N_dst, H_kv, 256]destinations (window rowjinto rowrow_map[j], int32[T_k], repeated rows accumulate), or returning(dk_win, dv_win)stores the FP32 window gradients[T_k, H_kv, 256]into them;k/vthen get no gradient. Nothing is held for the backward.max_seqlen_k (int, optional) – A host int at least every segment’s key count (the caller’s contract, as FlashAttention’s
max_seqlen_k; a smaller value leaves key blocks out); it sizes the backward grid.Nonebounds it byT_k(always correct; more CTAs that retire at once). Never read from the device.workspace (optional) – A workspace of
cake_gqa_det_train_workspace(), reused across calls and shapes (grown, never shrunk; one stream at a time); autograd holds it by weak reference.None: the device’s per-stream default workspace.return_lse (bool) – Also return the FP32 natural-log LSE
[H_q, T_q](no gradient flows through it).cu_seqlens_q_host (sequence of int or torch.Tensor, optional) – Host copies of
cu_seqlens_q/cu_seqlens_k(lists or 1-D CPU tensors; both or neither; never copied to the device). With them the backward plans its dS^T workspace chunks from each key block’s exact tile count: fewer, fuller chunks on long layouts and a workspace of the exact size; without them it bounds the counts fromT_qandB. The kernels read the device boundaries: host boundaries that disagree only move the chunks (and the dQ rounding) while every device key segment fits the host plan’s key blocks and every chunk holds its device tiles; otherwise a device-side assert fires before the main kernel runs (reported at the next synchronization; the CUDA context is unusable after it). The dQ bits depend on them.cu_seqlens_k_host (sequence of int or torch.Tensor, optional) – Host copies of
cu_seqlens_q/cu_seqlens_k(lists or 1-D CPU tensors; both or neither; never copied to the device). With them the backward plans its dS^T workspace chunks from each key block’s exact tile count: fewer, fuller chunks on long layouts and a workspace of the exact size; without them it bounds the counts fromT_qandB. The kernels read the device boundaries: host boundaries that disagree only move the chunks (and the dQ rounding) while every device key segment fits the host plan’s key blocks and every chunk holds its device tiles; otherwise a device-side assert fires before the main kernel runs (reported at the next synchronization; the CUDA context is unusable after it). The dQ bits depend on them.
- Returns:
outBF16[T_q, H_q, 256], or(out, lse).- Return type:
Union[torch.Tensor, tuple[torch.Tensor, torch.Tensor]]
Notes
The forward and the backward are fixed-order:
out,lseand every gradient repeat bit for bit run to run (one shape, onemax_seqlen_k, one workspace budget, the same host boundaries); neither raises PyTorch’s determinism alert.