flashinfer.gdn_prefill.chunk_gated_delta_rule

flashinfer.gdn_prefill.chunk_gated_delta_rule(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, state_checkpoints: Tensor | None = None, checkpoint_cu_starts: Tensor | None = None, checkpoint_every_n_tokens: int = 0, use_cp: Literal['auto'] | bool = 'auto', state_indices: Tensor | None = None, _cp_chunk_len: int | None = None, backend: Literal['auto', 'flashinfer', 'cake_gdn', 'cudnn', 'sm120_cute'] = 'auto', max_seqlen: int | None = None, *, gate_domain: Literal['linear', 'log'] = 'linear', use_gate_in_kernel: bool = False, A_log: Tensor | None = None, dt_bias: Tensor | None = None, beta_is_logit: bool = False, cu_seqlens_cpu: Sequence[int] | Tensor | None = None, output_state_indices: Tensor | None = None, checkpoint_state_indices: Tensor | None = None, prefix_lens: Tensor | None = None) → Tensor | Tuple[Tensor, Tensor]

Chunked Gated Delta Rule (GDN) attention for prefill.

Implements the gated delta rule linear attention mechanism for efficient training and inference. Supports both GQA (grouped query attention) and GVA (grouped value attention) configurations.

Parameters:
  • q (torch.Tensor) – Queries of shape [total_seq_len, num_q_heads, head_size]. Must be contiguous and on CUDA.

  • k (torch.Tensor) – Keys of shape [total_seq_len, num_k_heads, head_size]. Must be contiguous and on CUDA.

  • v (torch.Tensor) – Values of shape [total_seq_len, num_v_heads, head_size]. Must be contiguous and on CUDA.

  • g (torch.Tensor, optional) – Forget gate (alpha) of shape [total_seq_len, num_sab_heads] where num_sab_heads = max(num_q_heads, num_v_heads). Must be float32 for precomputed alpha; raw gates may also be float16 or bfloat16 when use_gate_in_kernel=True. Defaults to all ones when None.

  • beta (torch.Tensor, optional) – Update gate (beta) of shape [total_seq_len, num_sab_heads]. Must be float32 for precomputed beta; logits may also use the Q dtype when beta_is_logit=True. Defaults to all ones when None.

  • scale (float, optional) – Scale factor for the attention scores. Defaults to 1 / sqrt(head_size) when None.

  • initial_state (torch.Tensor, optional) – Initial KV state. Packed, sequence-ordered shape [num_seqs, num_sab_heads, head_size, head_size]. Must be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Starts from zero state when None. When state_indices is given (SM8x/SM90/SM100/SM103/SM120), this is instead the state pool [N_pool, num_sab_heads, head_size, head_size] and sequence i reads its initial state from row state_indices[i]; on SM100/SM103 every rank-4 positive, non-overlapping stride layout is accepted.

  • output_final_state (bool) – Whether to output the final state. Default: False.

  • cu_seqlens (torch.Tensor) – Cumulative sequence lengths of shape [num_seqs + 1], integer int32 or int64 dtype on the same CUDA device as q. Required for variable-length sequences (varlen mode); must not be None. Repeated adjacent offsets represent legal zero-length sequences.

  • use_qk_l2norm_in_kernel (bool) – Whether to L2-normalize each Q/K head with epsilon 1e-6. Normalization accumulates in float32 and rounds back to the input dtype before the chunked kernel. Default: False.

  • output (torch.Tensor, optional) – Pre-allocated output tensor of shape [total_seq_len, num_o_heads, head_size] where num_o_heads = max(num_q_heads, num_v_heads). Allocated automatically when None.

  • output_state (torch.Tensor, optional) – Pre-allocated output state tensor. Packed, sequence-ordered shape [num_seqs, num_sab_heads, head_size, head_size]. May be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Required when output_final_state=True. When state_indices is given it is instead the output state pool [N_pool, ...] and sequence i’s final state is written to row state_indices[i], or to row output_state_indices[i] when that is given (in place when output_state is initial_state); it must be provided by the caller (auto-allocation is rejected, since a compact [num_seqs, ...] buffer would be indexed out of bounds by the pool slot ids). SM100/SM103 accepts the same positive, non-overlapping rank-4 stride layouts as initial_state.

  • state_checkpoints (torch.Tensor, optional) – Pre-allocated checkpoint tensor of shape [total_checkpoints, num_sab_heads, head_size, head_size]. May be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Required when checkpoint_every_n_tokens > 0. Context-parallel checkpointing is currently supported on SM8x, SM90, SM100, and SM120.

  • checkpoint_cu_starts (torch.Tensor, optional) – Cumulative checkpoint counts of shape [num_seqs + 1], int32 or int64 on the same CUDA device as q. checkpoint_cu_starts[i+1] - checkpoint_cu_starts[i] is the number of checkpoints for sequence i: seq_len_i // checkpoint_every_n_tokens, or (prefix_lens[i] + seq_len_i) // N - prefix_lens[i] // N with prefix_lens (N = checkpoint_every_n_tokens). With checkpoint_state_indices only checkpoint_cu_starts[i] is read and [num_seqs] entries are also accepted. Required when checkpoint_every_n_tokens > 0. Without checkpoint_state_indices the values must be monotonic and consistent with cu_seqlens; this caller precondition is not checked at launch to avoid a device-to-host synchronization.

  • checkpoint_every_n_tokens (int) – Store intermediate state every N tokens. Must be a multiple of the chunk size (64). 0 disables checkpointing (default).

  • use_cp (Literal["auto"] | bool, optional:) – Whether to use the architecture-specific context-parallel DSL implementation on SM8x, SM90, SM100/SM103/SM107 or SM120, where it is supported and low-parallelism heuristics match. SM100/SM103/SM107 uses the generated GDN CP-only four-stage implementation for structurally supported shapes; other legal configurations retain the CuTe-DSL implementation. "auto" enables conservative routing, True requires CP support, and False disables CP. Default: "auto".

  • state_indices (torch.Tensor, optional) –

    Int32 or int64 tensor of shape [num_seqs] (SM8x/SM90/SM100/SM103/SM120). When provided, initial_state and output_state are treated as a state pool whose first dimension is indexed by these slot ids rather than laid out in sequence order: sequence i reads its initial state from row state_indices[i] and writes its final state back to the same row (in place when output_state is initial_state), or to row output_state_indices[i] when that is given. This lets callers that keep a paged/indexed state pool avoid gathering the active rows into a packed buffer and scattering the result back. The pool may be any positive, non-overlapping rank-4 stride layout on SM100/SM103. None (default) keeps sequence-ordered row mapping without requiring the physical view itself to be contiguous.

    Without output_state_indices the ids must be unique: as with any indexed scatter, two sequences sharing a slot id would concurrently write the same pool row across work tiles, leaving that row’s final state nondeterministic. Uniqueness is a caller precondition (not checked at launch, to avoid a per-call host sync); the caller’s slot allocator is expected to guarantee it. With output_state_indices read ids may repeat or be -1; only the rows a sequence writes must not be read or written by another sequence. backend="cudnn" requires frontend 1.31+, int32 indices and float32/bfloat16 pools with dense [H, V, K] rows and a 16-byte aligned slot stride. It supports identical input/output pool views or disjoint memory, and does not support checkpoints.

  • _cp_chunk_len (int, optional) – Internal context-parallel chunk-length override used for testing and tuning. None lets the CP backend select the length automatically; an explicit value must be a multiple of 64.

  • backend ({"auto", "flashinfer", "cake_gdn", "cudnn", "sm120_cute"}) – auto uses the native SM90/SM100/SM120 kernels and context-parallel routing, and may select cuDNN for measured SM100 raw-gate inference workloads. Automatic cuDNN selection requires both gate-fusion flags, BF16 inputs/gates, caller-normalized Q/K, separate compact float32 input/output states with output_final_state=True, and a caller-owned output. Explicit CP, pools and checkpoints retain native routing. Requires cuDNN frontend 1.31 GA and CuTe DSL 4.7 or newer. Cake kernels require an explicit cake_gdn request. Use backend="cake_gdn", use_cp=True for Cake CP on SM100/SM103/SM107; use_cp=False retains Cake non-CP, and "auto" enters Cake CP only inside the measured SM100/SM103 band (BF16/FP16 I/O, FP32 initial and final state, gates, no checkpoints or state indices, 4 * num_seqs * max(Hq, Hv) < SMs and max_seqlen >= 4096; without a max_seqlen hint the band estimates the longest sequence as ceil(total_seq_len / num_seqs)) or the SM107 bands (FP16 I/O; or BF16 I/O for one sequence of at most 65536 tokens, a multiple of 128, with int32 cu_seqlens, FP32 initial and final state, gates, caller-normalized Q/K and the measured head geometries, where the non-CP kernel fails closed). Explicit Cake requests fail for unsupported inputs without falling back to another backend. The Cake non-CP route performs no device-to-host synchronization and no eager warm-up call before CUDA Graph capture (the one-time kernel compile/load per variant is the only eager step); see cu_seqlens_cpu. The explicit cake_gdn non-CP route serves head_size == 128 with BF16 or FP16 I/O (one dtype for Q/K/V/output, caller-normalized Q/K, FP32 gates). State dtypes: FP32 with either I/O dtype; BF16, FP16 and FP8 (e4m3/e5m2) with BF16 I/O, the low-precision three only with an initial and a final state on the DV-split schedule. The initial state may be zero (None) or given and the final state is optional; state_indices pools need both. Checkpoints are limited to the frozen FP16-I/O FP32-state packed contract without an initial state and to the SGLang TP4 (H=4, HV=8) BF16-I/O BF16-state indexed DV-split contract at 64-token intervals. State rows must be dense [H, 128, 128] (strides (stride(0), 16384, 128, 1)); the data pointers of initial_state, output_state and state_checkpoints and the slot stride stride(0) of initial_state and output_state need only be element-aligned, not 16-byte aligned (sliced, offset or padded state pools give the same bits as 16-byte-aligned buffers): 16-byte-aligned state runs the unguarded body, element-aligned state a bit-identical guarded twin. The frozen (HEAD_GROUP_LOG2, IS_GQA, NUM_O_HEADS_LOG2) geometry set of each contract is enumerated by flashinfer.jit.cake_gdn.cake_gdn_prefill_geometries() from csrc/gdn/cake/manifest.json; on SM107 a DV-split key whose geometry only the full-DV set lists runs the full-DV record, and an unlisted geometry, schedule or architecture raises CakeGDNUnsupportedError. The BF16-I/O FP32-state initial+final contract covers the Qwen3-Next / Qwen3.5 per-rank geometries for TP1, TP2, TP4 and TP8 in both physical schedules (DV-split and full-DV). cudnn runs cuDNN’s fused SM100 linear-attention engine through flashinfer.cudnn.cudnn_chunk_gated_delta_rule().

  • max_seqlen (int, optional) – Safe upper bound on the maximum logical sequence length, no larger than total_seq_len. CP kernels use this host-side hint to bound their per-sequence launch grids without reading cu_seqlens back from the GPU. When omitted, CP uses total_seq_len, which is correct for any batch; passing the exact maximum of a batched call lets CP launch smaller grids. The SM100/SM103 use_cp="auto" band reads the hint (or the exact maximum from cu_seqlens_cpu) as the longest sequence and, without either, the balanced-batch estimate ceil(total_seq_len / num_seqs); pass the exact maximum so the band decision matches the Cake dispatcher’s. backend="cake_gdn" without CP uses the same bound (or the exact maximum from cu_seqlens_cpu) to select its kernel variant; a loose bound is always correct and only affects whether the single-chunk kernel is eligible. The persistent grid does not depend on the bound: it follows the per-sequence lengths of cu_seqlens_cpu when given (see below), else the residency rule.

  • gate_domain ({"linear", "log"}) – Domain of precomputed g: linear alpha (default) or natural-log alpha. None still means no decay. Ignored when use_gate_in_kernel is set, since that flag makes g a raw gate input.

  • use_gate_in_kernel (bool) – Interpret g as raw logits and compute log(alpha) = -exp(A_log) * softplus(g + dt_bias). cuDNN fuses this transform; other backends materialize the same FP32 alpha.

  • A_log (torch.Tensor, optional) – Per-head [num_sab_heads] parameters, required with use_gate_in_kernel. Float32, float16 or bfloat16, on q’s device.

  • dt_bias (torch.Tensor, optional) – Per-head [num_sab_heads] parameters, required with use_gate_in_kernel. Float32, float16 or bfloat16, on q’s device.

  • beta_is_logit (bool) – Apply sigmoid to beta in FP32, round to beta’s input dtype, then accumulate in FP32. cuDNN fuses the transform; other backends materialize it. The default accepts precomputed beta.

  • cu_seqlens_cpu (Sequence[int] or torch.Tensor, optional) – Host copy of cu_seqlens: a one-dimensional int32/int64 CPU tensor or a sequence of Python ints of length num_seqs + 1 holding the same values (vLLM’s query_start_loc_cpu, the prefix sums of SGLang’s extend_seq_lens_cpu). Used by backend="cake_gdn" without CP, and on SM8x by the use_cp="auto" and V32 rules, which need the exact longest sequence and keep the fused path without it. It is also required by the explicit experimental sm120_cute route. When given, the Cake route validates it on the host (non-decreasing, first entry 0, last entry total_seq_len; a mismatch raises before any launch), takes the exact longest sequence as the bound for variant selection, checks the checkpoint count exactly, and enables the ragged-batch launch grid: from the per-sequence lengths the host models the persistent kernel’s strided tile assignment against one CTA per tile and launches one CTA per tile when the modelled critical path shrinks by at least 4 % (same kernel binary, bitwise identical outputs); equal-length batches keep the residency grid. When omitted the route still reads nothing back from the device: the bound is max_seqlen (or total_seq_len), the persistent grid follows the residency rule alone and cu_seqlens is resolved by the kernel at every launch, which clamps every offset into [0, total_seq_len] and treats a non-monotonic pair as an empty sequence – a documented no-op, never an out-of-bounds access (checkpoint_cu_starts consistency stays a caller precondition). Int32 cu_seqlens reaches the kernel as is; int64 is cast on the stream at every call, except on records that read int64 metadata directly (every Cake non-CP prefill record on CC 10.7), where int64 reaches the kernel as is too. Nothing is cached by tensor identity, so a buffer rewritten in place is always read live, including on CUDA Graph replay.

  • output_state_indices (torch.Tensor, optional) – Contiguous int32 or int64 tensor of shape [num_seqs] on the device of q. Requires state_indices, an explicit output_state pool and output_final_state=True. Sequence i then reads its initial state from row state_indices[i] of initial_state and writes its final state to row output_state_indices[i] of output_state, which may be the same pool. A read id of -1 starts the sequence from the zero state; a write id of -1 skips its final-state write. A zero-length sequence writes nothing: its write row keeps its contents. Rows written by one sequence must not be read or written by another sequence of the same call.

  • checkpoint_state_indices (torch.Tensor, optional) – Contiguous int32 or int64 destination rows for checkpoints, on the device of q. Requires checkpoint_every_n_tokens > 0. state_checkpoints is then a state pool [N_pool, num_sab_heads, head_size, head_size] whose rows are dense [H, V, K] blocks (the row stride may be padded), so a pool it shares with initial_state/output_state needs that layout too. The j-th checkpoint of sequence i is written to row checkpoint_state_indices[checkpoint_cu_starts[i] + j]; a row of -1 skips that checkpoint. checkpoint_cu_starts ([num_seqs] or [num_seqs + 1] entries) only gives each sequence’s first entry, so a paged caller can pass its page table directly (see flashinfer.gdn_paged.chunk_gated_delta_rule_paged()).

  • prefix_lens (torch.Tensor, optional) –

    Contiguous int32 or int64 tensor of shape [num_seqs] on the device of q: the number of tokens of each sequence processed before this call. Requires checkpoint_every_n_tokens > 0. Checkpoints are then taken at absolute token positions: sequence i stores a checkpoint after its n-th new token whenever prefix_lens[i] + n is a multiple of checkpoint_every_n_tokens (1 <= n <= seq_len_i). Prefix lengths need not be multiples of the 64-token chunk; a sequence whose prefix is not chunk-aligned is chunked from its aligned absolute position, so every checkpoint falls on a chunk boundary. Sequence i then has (prefix_lens[i] + seq_len_i) // N - prefix_lens[i] // N checkpoints (N = checkpoint_every_n_tokens); size checkpoint_cu_starts accordingly.

    output_state_indices, checkpoint_state_indices and prefix_lens are served by the native non-CP kernels on SM8x, SM90, SM100/SM103/SM107 and SM120 (backend="auto" or "flashinfer"); use_cp=True, backend="cake_gdn" and backend="cudnn" reject them.

Returns:

When output_final_state=False, the output tensor of shape [total_seq_len, num_o_heads, head_size]. Otherwise a tuple (output, final_state) where final_state has shape [num_seqs, num_sab_heads, head_size, head_size] — or, when state_indices is given, the state pool [N_pool, ...] itself (i.e. output_state), whose rows named by state_indices (by output_state_indices when given; -1 rows are not written) now hold the updated final states.

Return type:

torch.Tensor or Tuple[torch.Tensor, torch.Tensor]

Notes

  • backend="sm120_cute" explicitly selects the experimental RTX 6000D SM120 D=128 CuTe implementation. Fast calls require native finite FP32 log gates (gate_domain="log"), FP32 beta/state and matching immutable cu_seqlens_cpu metadata. Unsupported calls, including checkpoints, state pools, linear/raw gates and low-precision state, use FlashInfer’s official implementation. Stable automatic dispatch never selects it.

  • Supports GQA (num_q_heads > num_k_heads = num_v_heads) and GVA (num_v_heads > num_q_heads = num_k_heads).

  • The final state layout is [N, H, V, K].

  • Requires SM8x (Ampere), SM90 (Hopper), SM100/SM103 (Blackwell), SM107 or SM120. The SM100 path requires head_size == 128. On SM100/SM103, gdn_cp supports structurally legal equal-head, GQA, and GVA shapes on CUDA 12.8, CUDA 12.9, and CUDA 13; SM107 supports the same shapes with an nvcc that can emit compute_107a. Other SM100 CP DSL routes require CUDA 13 and nvidia-cutlass-dsl[cu13]>=4.4.2 (pip install flashinfer-python[cu13]).

  • On SM8x the state may not be FP8.

  • use_cp="auto" selects CP on compute capability 8.0 when cu_seqlens_cpu gives a longest sequence of at least 8192 tokens and num_seqs * max(num_q_heads, num_v_heads) is at most 16.