flashinfer.gdp_prefill.chunk_gated_delta_product

flashinfer.gdp_prefill.chunk_gated_delta_product(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, num_householder: int = 1, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, *, backend: Literal['auto', 'cudnn', 'cake'] = 'auto', state_indices: Tensor | None = None, cu_seqlens_cpu: Tensor | None = None) → Tensor | Tuple[Tensor, Tensor]

Chunked Gated DeltaProduct (GDP) attention for prefill.

GDP generalizes flashinfer.chunk_gated_delta_rule() from one beta-gated Householder update per token to n = num_householder of them, with one per-head scalar decay per token: the decay acts before the token’s updates and the readout follows the last one. Per real token, with sub-token rows t*n .. t*n + n - 1 of k / v / beta,

\[\begin{split}S &\mathrel{*}= \alpha_t \\ v^{new}_{t,j} &= \beta_{t,j} (v_{t,j} - S k_{t,j}), \quad S \mathrel{+}= v^{new}_{t,j} \otimes k_{t,j}, \quad j = 0 \ldots n - 1 \\ o_t &= \mathrm{scale} \cdot S q_t\end{split}\]

num_householder == 1 is exactly flashinfer.chunk_gated_delta_rule().

Only varlen (packed / THD) input is accepted, matching flashinfer.chunk_gated_delta_rule(): pass cu_seqlens over the real tokens and lay the tokens of all sequences end to end.

Parameters:
  • q (torch.Tensor) – [total_seq_len, num_q_heads, head_size], float16 or bfloat16, at real-token rows. Strides are honored, so only the innermost dimension has to be contiguous.

  • k (torch.Tensor) – [total_seq_len * num_householder, num_k_heads / num_v_heads, head_size], float16 or bfloat16, on the expanded sub-token timeline: the n Householder updates of token t occupy rows t*n .. t*n + n - 1. num_k_heads must equal num_q_heads or num_v_heads.

  • v (torch.Tensor) – [total_seq_len * num_householder, num_k_heads / num_v_heads, head_size], float16 or bfloat16, on the expanded sub-token timeline: the n Householder updates of token t occupy rows t*n .. t*n + n - 1. num_k_heads must equal num_q_heads or num_v_heads.

  • g (torch.Tensor, optional) – Per-head forget gate in linear space (alpha, elementwise in (0, 1]), shape [total_seq_len, num_sab_heads] at real-token rows, at float32, bfloat16 or float16. All-ones (no decay) when None.

  • beta (torch.Tensor, optional) – Per-head, per-Householder update gate [total_seq_len * num_householder, num_sab_heads], post-sigmoid, in float32 or q.dtype. All-ones when None.

  • num_householder (int) – Householder updates per token (n >= 1).

  • scale (float, optional) – Query scale; 1 / sqrt(head_size) when None or 0.0, matching flashinfer.chunk_gated_delta_rule().

  • initial_state (torch.Tensor, optional) – Incoming recurrent state [num_seqs, num_sab_heads, head_size, head_size], V-major ([..., V, K]), float32 or bfloat16. A zero state when None.

  • output_final_state (bool) – Return the outgoing recurrent state alongside the output.

  • cu_seqlens (torch.Tensor) – Cumulative real-token sequence lengths [num_seqs + 1], int32 or int64. Required.

  • use_qk_l2norm_in_kernel (bool) – Fuse the q/k L2 normalization into the kernel instead of expecting pre-normalized inputs.

  • output (torch.Tensor, optional) – Pre-allocated output [total_seq_len, num_o_heads, head_size] at real-token rows, written in place. Allocated internally when None.

  • output_state (torch.Tensor, optional) – Pre-allocated final-state buffer, written in place. Allocated internally when None and output_final_state is set. It must not alias initial_state without state_indices; the kernel splits one sequence across CTAs, so the CTA reading the incoming state would race the one writing the outgoing state.

  • backend (Literal["auto", "cudnn", "cake"], optional) – "auto" (default) and "cudnn" run cuDNN’s fused SM100 linear-attention engine through flashinfer.cudnn.cudnn_chunk_gated_delta_product(). "cake" runs the Cake GDP prefill kernel (flashinfer.cake_gdp_backend.cake_chunk_gated_delta_product()): head_size == 128 with a 64-wide value / state row, one head count for q, k and v, num_householder in 1..4, an SM90 device (H100 / H200); the argument contract is this function’s, with initial_state / output_state in float32 (a bfloat16 per-sequence initial_state is converted) and contiguous state pools. One Cake kernel launch per call; the split programs add a workspace clear and the item-table upload. With cu_seqlens_cpu given, no device-to-host synchronization is issued (the split rule reads the host copy); without it the sequence offsets are copied to the host once. g=None runs the kernel’s unit-gate program and beta=None the unit-beta program, from a zero state; g=None together with beta=None, a unit-gate / unit-beta call with an initial_state or a state_indices pool, and a device without an exported route raise ValueError naming what is exported.

  • cu_seqlens_cpu (torch.Tensor, optional) – The host copy of cu_seqlens (a CPU int32 / int64 1-D tensor of N + 1 entries, as vLLM keeps query_start_loc_cpu). The Cake backend reads the sequence lengths on the host for the kernel’s split rule of record (a long sequence is cut into items for the gate-saturation split program; a batch the rule leaves whole runs the exact program; the two agree bitwise). When it is None the Cake backend reads cu_seqlens.cpu(): one device-to-host synchronization, which CUDA Graph capture does not allow, so a graph-captured caller passes it. It is checked for device, rank, length and dtype, never compared with cu_seqlens value by value. The cuDNN backends accept and ignore it.

  • state_indices (torch.Tensor, optional) – Contiguous int32 [num_seqs] slots in initial_state and output_state pools, requiring cuDNN frontend 1.31+. Pools have shape [N_pool, H, V, K], dense inner rows, and a 16-byte aligned slot stride. Slot values must be unique and in range. When output_final_state=True, an explicit output pool is required; it may be the identical input view or disjoint memory. The returned state is the whole output pool, with unselected slots unchanged. When false, neither pool is modified. Values are not read on the CPU.

Returns:

output, or (output, final_state) when output_final_state.

Return type:

torch.Tensor or Tuple[torch.Tensor, torch.Tensor]

Note

Requires an SM100-family (Blackwell) device and cudnn-frontend 1.29+ with the cutedsl extra (pip install 'nvidia-cudnn-frontend[cutedsl]'). Everything finer – head dims, input dtypes, head-count relations – is the engine’s call: a graph it cannot serve is declined by cuDNN (the per-engine reason lands in the frontend’s log).