flashinfer.cudnn.cudnn_chunk_gated_delta_rule¶
- flashinfer.cudnn.cudnn_chunk_gated_delta_rule(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, batch_invariant: bool = False, state_indices: Tensor | None = None, *, gate_domain: str = 'linear', use_gate_in_kernel: bool = False, A_log: Tensor | None = None, dt_bias: Tensor | None = None, beta_is_logit: bool = False) Tensor | tuple[Tensor, Tensor]¶
Chunked Gated Delta Rule prefill on cuDNN’s fused SM100 engine.
Argument meanings match
flashinfer.chunk_gated_delta_rule().Requires cudnn-frontend 1.31.0+ with the
cutedslextra. Everything else the engine decides for itself: it declines a graph it cannot serve (the per-engine reason lands in the frontend’s log).- Parameters:
q (torch.Tensor) –
[total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.k (torch.Tensor) –
[total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.v (torch.Tensor) –
[total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.g (torch.Tensor, optional) – Per-head forget gate in linear space (
alpha = exp(log_g)), shape[total_seq_len, num_sab_heads], passed to cuDNN as is (gate_domain="linear") atg’s own dtype, which cuDNN reads at float32, bfloat16 or float16. All-ones whenNone.beta (torch.Tensor, optional) – Per-head update gate
[total_seq_len, num_sab_heads], post-sigmoid, in float32 orq.dtype. All-ones whenNone.scale (float, optional) – Query scale;
1 / sqrt(head_dim)whenNoneor0.0, matching the native GDN path.initial_state (torch.Tensor, optional) – State
[num_seqs, num_sab_heads, 128, 128], V-major, float32 or bfloat16. cuDNN uses the same layout, so these pass through untransposed andoutput_stateis written in place by the kernel. Withoutstate_indices,output_statemust not aliasinitial_state: the engines split one sequence across CTAs, so the chunk-0 CTA reading the incoming state would race the last-chunk CTA writing the outgoing one. Like the state-slot uniquenessstate_indicesrelies on, this is a caller precondition rather than a launch-time check.output_state (torch.Tensor, optional) – State
[num_seqs, num_sab_heads, 128, 128], V-major, float32 or bfloat16. cuDNN uses the same layout, so these pass through untransposed andoutput_stateis written in place by the kernel. Withoutstate_indices,output_statemust not aliasinitial_state: the engines split one sequence across CTAs, so the chunk-0 CTA reading the incoming state would race the last-chunk CTA writing the outgoing one. Like the state-slot uniquenessstate_indicesrelies on, this is a caller precondition rather than a launch-time check.output_final_state (bool) – Return the outgoing recurrent state alongside the output; when unset no state is written at all.
cu_seqlens (torch.Tensor) –
[num_seqs + 1]int32 or int64. Required.use_qk_l2norm_in_kernel (bool) – Normalize Q/K with additive epsilon
1e-6and round to the input dtype inside the frontend, matching the public GDN API.output (torch.Tensor, optional) – Pre-allocated
[total_seq_len, num_o_heads, 128], written in place by the kernel.batch_invariant (bool) – Disable the split-K partition so the reduction order, and hence the result, does not depend on how sequences are batched. Costs the parallelism split-K exists to create on few long sequences and saves its fixed scheduling cost on many short ones.
state_indices (torch.Tensor, optional) – Contiguous int32
[num_seqs]pool slots on q’s device, requiring cuDNN frontend 1.31+. Both states are pools[N_pool, H, V, K]with dense inner rows and a 16-byte aligned slot stride. Slot ids must be unique and in range; these device values are a caller precondition and are not read back. Returning state requires an explicit output pool of the same shape/dtype, either the identical input view (a safe in-place plan) or disjoint memory. Unselected rows remain untouched. No pool is written when output_final_state=False.gate_domain ({"linear", "log"}) – Precomputed decay domain, matching the public GDN API.
use_gate_in_kernel (bool) – Fuse
-exp(A_log) * softplus(g + dt_bias)from raw g logits. Requires per-head A_log and dt_bias; gate_domain then does not apply.A_log (torch.Tensor, optional) – Per-head
[num_sab_heads]raw-gate parameters on q’s device.dt_bias (torch.Tensor, optional) – Per-head
[num_sab_heads]raw-gate parameters on q’s device.beta_is_logit (bool) – Fuse sigmoid with rounding to beta’s input dtype before FP32 use.
- Returns:
output, or(output, final_state)whenoutput_final_state.- Return type:
torch.Tensor or Tuple[torch.Tensor, torch.Tensor]