flashinfer.cake_gqa_det_train.cake_gqa_det_train_backward¶
- flashinfer.cake_gqa_det_train.cake_gqa_det_train_backward(q: Tensor, k: Tensor, v: Tensor, out: Tensor, lse: Tensor, dout: Tensor, cu_seqlens_q: Tensor, cu_seqlens_k: Tensor, *, softmax_scale: float | None = None, dq: Tensor | None = None, dk: Tensor | None = None, dv: Tensor | None = None, dkv_dtype: dtype = torch.bfloat16, dk_acc: Tensor | None = None, dv_acc: Tensor | None = None, row_map: Tensor | None = None, max_seqlen_k: int | None = None, workspace: Any | None = None, cu_seqlens_q_host: Sequence[int] | Tensor | None = None, cu_seqlens_k_host: Sequence[int] | Tensor | None = None) tuple[Tensor, Tensor | None, Tensor | None]¶
Backward without autograd:
(dq, dk, dv)from the forward’soutandlse.dK / dV are FP32 sums over the query heads of each key head.
- Parameters:
q (torch.Tensor) – The forward’s operands (see
cake_gqa_det()).k (torch.Tensor) – The forward’s operands (see
cake_gqa_det()).v (torch.Tensor) – The forward’s operands (see
cake_gqa_det()).out (torch.Tensor) – The forward’s BF16
[T_q, H_q, 256]output.lse (torch.Tensor) – The forward’s FP32
[H_q, T_q]LSE.dout (torch.Tensor) – Contiguous BF16
[T_q, H_q, 256]gradient ofout.cu_seqlens_q (torch.Tensor) – As in
cake_gqa_det().cu_seqlens_k (torch.Tensor) – As in
cake_gqa_det().softmax_scale (float, optional) – The forward’s scale; default
256 ** -0.5.dq (torch.Tensor, optional) – A contiguous BF16
[T_q, H_q, 256]to write;Noneallocates one.dk (torch.Tensor, optional) – Contiguous
[T_k, H_kv, 256]tensors ofdkv_dtypeto write (overwritten);Noneallocates them. Not used withrow_map.dv (torch.Tensor, optional) – Contiguous
[T_k, H_kv, 256]tensors ofdkv_dtypeto write (overwritten);Noneallocates them. Not used withrow_map.dkv_dtype (torch.dtype) –
torch.bfloat16(default): dK / dV rounded once to BF16.torch.float32: the FP32 window gradients are stored.dk_acc (torch.Tensor, optional) – FP32
[N_dst, H_kv, 256]destinations that the window gradients are added into; they needrow_map.dv_acc (torch.Tensor, optional) – FP32
[N_dst, H_kv, 256]destinations that the window gradients are added into; they needrow_map.row_map (torch.Tensor, optional) – int32
[T_k]: window rowjadds into destination rowrow_map[j](rows may repeat; out-of-range rows are not checked).max_seqlen_k (int, optional) – As in
cake_gqa_det().workspace (optional) – As in
cake_gqa_det().cu_seqlens_q_host (sequence of int or torch.Tensor, optional) – As in
cake_gqa_det().cu_seqlens_k_host (sequence of int or torch.Tensor, optional) – As in
cake_gqa_det().
- Returns:
(dq, dk, dv); withrow_map,dk/dvaredk_acc/dv_acc.- Return type:
tuple[torch.Tensor, Optional[torch.Tensor], Optional[torch.Tensor]]
Notes
Bitwise reproducible run to run for one shape, one
max_seqlen_k, one workspacebudget_bytesand the same host boundaries (given or not): dK and dV of a key block are summed in tensor memory in one order, dS^T goes to the workspace and dQ is a GEMM over it with each output tile summed in one order, and the row-map destinations are folded in ascending window order. A call whose dS^T exceeds the budget runs in chunks (KV heads first, then key-block ranges); the budget and the host boundaries move the dQ rounding, not its accuracy. Raises no PyTorch determinism alert.