flashinfer.cudnn.cudnn_chunk_gated_delta_rule

flashinfer.cudnn.cudnn_chunk_gated_delta_rule(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, batch_invariant: bool = False, state_indices: Tensor | None = None, *, gate_domain: str = 'linear', use_gate_in_kernel: bool = False, A_log: Tensor | None = None, dt_bias: Tensor | None = None, beta_is_logit: bool = False) → Tensor | tuple[Tensor, Tensor]

Chunked Gated Delta Rule prefill on cuDNN’s fused SM100 engine.

Argument meanings match flashinfer.chunk_gated_delta_rule().

Requires cudnn-frontend 1.31.0+ with the cutedsl extra. Everything else the engine decides for itself: it declines a graph it cannot serve (the per-engine reason lands in the frontend’s log).

Parameters:
  • q (torch.Tensor) – [total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.

  • k (torch.Tensor) – [total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.

  • v (torch.Tensor) – [total_seq_len, num_q_heads / num_k_heads / num_v_heads, 128], packed. Strides are passed through to cuDNN, so only the innermost dim has to be contiguous.

  • g (torch.Tensor, optional) – Per-head forget gate in linear space (alpha = exp(log_g)), shape [total_seq_len, num_sab_heads], passed to cuDNN as is (gate_domain="linear") at g’s own dtype, which cuDNN reads at float32, bfloat16 or float16. All-ones when None.

  • beta (torch.Tensor, optional) – Per-head update gate [total_seq_len, num_sab_heads], post-sigmoid, in float32 or q.dtype. All-ones when None.

  • scale (float, optional) – Query scale; 1 / sqrt(head_dim) when None or 0.0, matching the native GDN path.

  • initial_state (torch.Tensor, optional) – State [num_seqs, num_sab_heads, 128, 128], V-major, float32 or bfloat16. cuDNN uses the same layout, so these pass through untransposed and output_state is written in place by the kernel. Without state_indices, output_state must not alias initial_state: the engines split one sequence across CTAs, so the chunk-0 CTA reading the incoming state would race the last-chunk CTA writing the outgoing one. Like the state-slot uniqueness state_indices relies on, this is a caller precondition rather than a launch-time check.

  • output_state (torch.Tensor, optional) – State [num_seqs, num_sab_heads, 128, 128], V-major, float32 or bfloat16. cuDNN uses the same layout, so these pass through untransposed and output_state is written in place by the kernel. Without state_indices, output_state must not alias initial_state: the engines split one sequence across CTAs, so the chunk-0 CTA reading the incoming state would race the last-chunk CTA writing the outgoing one. Like the state-slot uniqueness state_indices relies on, this is a caller precondition rather than a launch-time check.

  • output_final_state (bool) – Return the outgoing recurrent state alongside the output; when unset no state is written at all.

  • cu_seqlens (torch.Tensor) – [num_seqs + 1] int32 or int64. Required.

  • use_qk_l2norm_in_kernel (bool) – Normalize Q/K with additive epsilon 1e-6 and round to the input dtype inside the frontend, matching the public GDN API.

  • output (torch.Tensor, optional) – Pre-allocated [total_seq_len, num_o_heads, 128], written in place by the kernel.

  • batch_invariant (bool) – Disable the split-K partition so the reduction order, and hence the result, does not depend on how sequences are batched. Costs the parallelism split-K exists to create on few long sequences and saves its fixed scheduling cost on many short ones.

  • state_indices (torch.Tensor, optional) – Contiguous int32 [num_seqs] pool slots on q’s device, requiring cuDNN frontend 1.31+. Both states are pools [N_pool, H, V, K] with dense inner rows and a 16-byte aligned slot stride. Slot ids must be unique and in range; these device values are a caller precondition and are not read back. Returning state requires an explicit output pool of the same shape/dtype, either the identical input view (a safe in-place plan) or disjoint memory. Unselected rows remain untouched. No pool is written when output_final_state=False.

  • gate_domain ({"linear", "log"}) – Precomputed decay domain, matching the public GDN API.

  • use_gate_in_kernel (bool) – Fuse -exp(A_log) * softplus(g + dt_bias) from raw g logits. Requires per-head A_log and dt_bias; gate_domain then does not apply.

  • A_log (torch.Tensor, optional) – Per-head [num_sab_heads] raw-gate parameters on q’s device.

  • dt_bias (torch.Tensor, optional) – Per-head [num_sab_heads] raw-gate parameters on q’s device.

  • beta_is_logit (bool) – Fuse sigmoid with rounding to beta’s input dtype before FP32 use.

Returns:

output, or (output, final_state) when output_final_state.

Return type:

torch.Tensor or Tuple[torch.Tensor, torch.Tensor]