flashinfer.gdp_prefill.chunk_gated_delta_product¶
- flashinfer.gdp_prefill.chunk_gated_delta_product(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, num_householder: int = 1, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, *, backend: Literal['auto', 'cudnn', 'cake'] = 'auto', state_indices: Tensor | None = None, cu_seqlens_cpu: Tensor | None = None) Tensor | Tuple[Tensor, Tensor]¶
Chunked Gated DeltaProduct (GDP) attention for prefill.
GDP generalizes
flashinfer.chunk_gated_delta_rule()from one beta-gated Householder update per token ton = num_householderof them, with one per-head scalar decay per token: the decay acts before the token’s updates and the readout follows the last one. Per real token, with sub-token rowst*n .. t*n + n - 1ofk/v/beta,\[\begin{split}S &\mathrel{*}= \alpha_t \\ v^{new}_{t,j} &= \beta_{t,j} (v_{t,j} - S k_{t,j}), \quad S \mathrel{+}= v^{new}_{t,j} \otimes k_{t,j}, \quad j = 0 \ldots n - 1 \\ o_t &= \mathrm{scale} \cdot S q_t\end{split}\]num_householder == 1is exactlyflashinfer.chunk_gated_delta_rule().Only varlen (packed / THD) input is accepted, matching
flashinfer.chunk_gated_delta_rule(): passcu_seqlensover the real tokens and lay the tokens of all sequences end to end.- Parameters:
q (torch.Tensor) –
[total_seq_len, num_q_heads, head_size], float16 or bfloat16, at real-token rows. Strides are honored, so only the innermost dimension has to be contiguous.k (torch.Tensor) –
[total_seq_len * num_householder, num_k_heads / num_v_heads, head_size], float16 or bfloat16, on the expanded sub-token timeline: thenHouseholder updates of tokentoccupy rowst*n .. t*n + n - 1.num_k_headsmust equalnum_q_headsornum_v_heads.v (torch.Tensor) –
[total_seq_len * num_householder, num_k_heads / num_v_heads, head_size], float16 or bfloat16, on the expanded sub-token timeline: thenHouseholder updates of tokentoccupy rowst*n .. t*n + n - 1.num_k_headsmust equalnum_q_headsornum_v_heads.g (torch.Tensor, optional) – Per-head forget gate in linear space (
alpha, elementwise in(0, 1]), shape[total_seq_len, num_sab_heads]at real-token rows, at float32, bfloat16 or float16. All-ones (no decay) whenNone.beta (torch.Tensor, optional) – Per-head, per-Householder update gate
[total_seq_len * num_householder, num_sab_heads], post-sigmoid, in float32 orq.dtype. All-ones whenNone.num_householder (int) – Householder updates per token (
n >= 1).scale (float, optional) – Query scale;
1 / sqrt(head_size)whenNoneor0.0, matchingflashinfer.chunk_gated_delta_rule().initial_state (torch.Tensor, optional) – Incoming recurrent state
[num_seqs, num_sab_heads, head_size, head_size], V-major ([..., V, K]), float32 or bfloat16. A zero state whenNone.output_final_state (bool) – Return the outgoing recurrent state alongside the output.
cu_seqlens (torch.Tensor) – Cumulative real-token sequence lengths
[num_seqs + 1], int32 or int64. Required.use_qk_l2norm_in_kernel (bool) – Fuse the q/k L2 normalization into the kernel instead of expecting pre-normalized inputs.
output (torch.Tensor, optional) – Pre-allocated output
[total_seq_len, num_o_heads, head_size]at real-token rows, written in place. Allocated internally whenNone.output_state (torch.Tensor, optional) – Pre-allocated final-state buffer, written in place. Allocated internally when
Noneandoutput_final_stateis set. It must not aliasinitial_statewithoutstate_indices; the kernel splits one sequence across CTAs, so the CTA reading the incoming state would race the one writing the outgoing state.backend (Literal["auto", "cudnn", "cake"], optional) –
"auto"(default) and"cudnn"run cuDNN’s fused SM100 linear-attention engine throughflashinfer.cudnn.cudnn_chunk_gated_delta_product()."cake"runs the Cake GDP prefill kernel (flashinfer.cake_gdp_backend.cake_chunk_gated_delta_product()):head_size == 128with a 64-wide value / state row, one head count for q, k and v,num_householderin 1..4, an SM90 device (H100 / H200); the argument contract is this function’s, withinitial_state/output_statein float32 (a bfloat16 per-sequenceinitial_stateis converted) and contiguous state pools. One Cake kernel launch per call; the split programs add a workspace clear and the item-table upload. Withcu_seqlens_cpugiven, no device-to-host synchronization is issued (the split rule reads the host copy); without it the sequence offsets are copied to the host once.g=Noneruns the kernel’s unit-gate program andbeta=Nonethe unit-beta program, from a zero state;g=Nonetogether withbeta=None, a unit-gate / unit-beta call with aninitial_stateor astate_indicespool, and a device without an exported route raiseValueErrornaming what is exported.cu_seqlens_cpu (torch.Tensor, optional) – The host copy of
cu_seqlens(a CPU int32 / int64 1-D tensor ofN + 1entries, as vLLM keepsquery_start_loc_cpu). The Cake backend reads the sequence lengths on the host for the kernel’s split rule of record (a long sequence is cut into items for the gate-saturation split program; a batch the rule leaves whole runs the exact program; the two agree bitwise). When it is None the Cake backend readscu_seqlens.cpu(): one device-to-host synchronization, which CUDA Graph capture does not allow, so a graph-captured caller passes it. It is checked for device, rank, length and dtype, never compared withcu_seqlensvalue by value. The cuDNN backends accept and ignore it.state_indices (torch.Tensor, optional) – Contiguous int32
[num_seqs]slots ininitial_stateandoutput_statepools, requiring cuDNN frontend 1.31+. Pools have shape[N_pool, H, V, K], dense inner rows, and a 16-byte aligned slot stride. Slot values must be unique and in range. Whenoutput_final_state=True, an explicit output pool is required; it may be the identical input view or disjoint memory. The returned state is the whole output pool, with unselected slots unchanged. When false, neither pool is modified. Values are not read on the CPU.
- Returns:
output, or(output, final_state)whenoutput_final_state.- Return type:
torch.Tensor or Tuple[torch.Tensor, torch.Tensor]
Note
Requires an SM100-family (Blackwell) device and cudnn-frontend 1.29+ with the
cutedslextra (pip install 'nvidia-cudnn-frontend[cutedsl]'). Everything finer – head dims, input dtypes, head-count relations – is the engine’s call: a graph it cannot serve is declined by cuDNN (the per-engine reason lands in the frontend’s log).