flashinfer.cake_gqa_det_train.cake_gqa_det

flashinfer.cake_gqa_det_train.cake_gqa_det(q: Tensor, k: Tensor, v: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, softmax_scale: float | None = None, causal: bool = True, dkv_acc: Callable[[], tuple] | None = None, max_seqlen_k: int | None = None, workspace: Any | None = None, return_lse: bool = False, cu_seqlens_q_host: Sequence[int] | Tensor | None = None, cu_seqlens_k_host: Sequence[int] | Tensor | None = None) → Tensor | tuple[Tensor, Tensor]

Packed-varlen causal GQA attention, head dimension 256, with autograd (CakeGQADetFunction).

Parameters:
  • q (torch.Tensor) – BF16 [T_q, H_q, 256] queries.

  • k (torch.Tensor) – BF16 [T_k, H_kv, 256] keys and values; query head h reads key head h / (H_q / H_kv).

  • v (torch.Tensor) – BF16 [T_k, H_kv, 256] keys and values; query head h reads key head h / (H_q / H_kv).

  • cu_seqlens_q (torch.Tensor) – int32 [B + 1] on the device, cu[0] = 0, non-decreasing, cu_q[B] = T_q, cu_k[B] = T_k.

  • cu_seqlens_k (torch.Tensor) – int32 [B + 1] on the device, cu[0] = 0, non-decreasing, cu_q[B] = T_q, cu_k[B] = T_k.

  • softmax_scale (float, optional) – Default 256 ** -0.5.

  • causal (bool) – Must be True (bottom-right alignment per segment).

  • dkv_acc (callable, optional) – None: k / v get BF16 gradients (FP32 sums over the query heads rounded once). A zero-argument callable (CakeGQADetDkvAccSource), called in backward, returning (dk_acc, dv_acc, row_map) adds the FP32 window gradients into the caller’s FP32 [N_dst, H_kv, 256] destinations (window row j into row row_map[j], int32 [T_k], repeated rows accumulate), or returning (dk_win, dv_win) stores the FP32 window gradients [T_k, H_kv, 256] into them; k / v then get no gradient. Nothing is held for the backward.

  • max_seqlen_k (int, optional) – A host int at least every segment’s key count (the caller’s contract, as FlashAttention’s max_seqlen_k; a smaller value leaves key blocks out); it sizes the backward grid. None bounds it by T_k (always correct; more CTAs that retire at once). Never read from the device.

  • workspace (optional) – A workspace of cake_gqa_det_train_workspace(), reused across calls and shapes (grown, never shrunk; one stream at a time); autograd holds it by weak reference. None: the device’s per-stream default workspace.

  • return_lse (bool) – Also return the FP32 natural-log LSE [H_q, T_q] (no gradient flows through it).

  • cu_seqlens_q_host (sequence of int or torch.Tensor, optional) – Host copies of cu_seqlens_q / cu_seqlens_k (lists or 1-D CPU tensors; both or neither; never copied to the device). With them the backward plans its dS^T workspace chunks from each key block’s exact tile count: fewer, fuller chunks on long layouts and a workspace of the exact size; without them it bounds the counts from T_q and B. The kernels read the device boundaries: host boundaries that disagree only move the chunks (and the dQ rounding) while every device key segment fits the host plan’s key blocks and every chunk holds its device tiles; otherwise a device-side assert fires before the main kernel runs (reported at the next synchronization; the CUDA context is unusable after it). The dQ bits depend on them.

  • cu_seqlens_k_host (sequence of int or torch.Tensor, optional) – Host copies of cu_seqlens_q / cu_seqlens_k (lists or 1-D CPU tensors; both or neither; never copied to the device). With them the backward plans its dS^T workspace chunks from each key block’s exact tile count: fewer, fuller chunks on long layouts and a workspace of the exact size; without them it bounds the counts from T_q and B. The kernels read the device boundaries: host boundaries that disagree only move the chunks (and the dQ rounding) while every device key segment fits the host plan’s key blocks and every chunk holds its device tiles; otherwise a device-side assert fires before the main kernel runs (reported at the next synchronization; the CUDA context is unusable after it). The dQ bits depend on them.

Returns:

out BF16 [T_q, H_q, 256], or (out, lse).

Return type:

Union[torch.Tensor, tuple[torch.Tensor, torch.Tensor]]

Notes

The forward and the backward are fixed-order: out, lse and every gradient repeat bit for bit run to run (one shape, one max_seqlen_k, one workspace budget, the same host boundaries); neither raises PyTorch’s determinism alert.