flashinfer.gdn_prefill.chunk_gated_delta_rule¶
- flashinfer.gdn_prefill.chunk_gated_delta_rule(q: Tensor, k: Tensor, v: Tensor, g: Tensor | None = None, beta: Tensor | None = None, scale: float | None = None, initial_state: Tensor | None = None, output_final_state: bool = False, cu_seqlens: Tensor | None = None, use_qk_l2norm_in_kernel: bool = False, output: Tensor | None = None, output_state: Tensor | None = None, state_checkpoints: Tensor | None = None, checkpoint_cu_starts: Tensor | None = None, checkpoint_every_n_tokens: int = 0, use_cp: Literal['auto'] | bool = 'auto', state_indices: Tensor | None = None, _cp_chunk_len: int | None = None, backend: Literal['auto', 'flashinfer', 'cake_gdn', 'cudnn', 'sm120_cute'] = 'auto', max_seqlen: int | None = None, *, gate_domain: Literal['linear', 'log'] = 'linear', use_gate_in_kernel: bool = False, A_log: Tensor | None = None, dt_bias: Tensor | None = None, beta_is_logit: bool = False, cu_seqlens_cpu: Sequence[int] | Tensor | None = None, output_state_indices: Tensor | None = None, checkpoint_state_indices: Tensor | None = None, prefix_lens: Tensor | None = None) Tensor | Tuple[Tensor, Tensor]¶
Chunked Gated Delta Rule (GDN) attention for prefill.
Implements the gated delta rule linear attention mechanism for efficient training and inference. Supports both GQA (grouped query attention) and GVA (grouped value attention) configurations.
- Parameters:
q (torch.Tensor) – Queries of shape
[total_seq_len, num_q_heads, head_size]. Must be contiguous and on CUDA.k (torch.Tensor) – Keys of shape
[total_seq_len, num_k_heads, head_size]. Must be contiguous and on CUDA.v (torch.Tensor) – Values of shape
[total_seq_len, num_v_heads, head_size]. Must be contiguous and on CUDA.g (torch.Tensor, optional) – Forget gate (alpha) of shape
[total_seq_len, num_sab_heads]wherenum_sab_heads = max(num_q_heads, num_v_heads). Must be float32 for precomputed alpha; raw gates may also be float16 or bfloat16 whenuse_gate_in_kernel=True. Defaults to all ones whenNone.beta (torch.Tensor, optional) – Update gate (beta) of shape
[total_seq_len, num_sab_heads]. Must be float32 for precomputed beta; logits may also use the Q dtype whenbeta_is_logit=True. Defaults to all ones whenNone.scale (float, optional) – Scale factor for the attention scores. Defaults to
1 / sqrt(head_size)whenNone.initial_state (torch.Tensor, optional) – Initial KV state. Packed, sequence-ordered shape
[num_seqs, num_sab_heads, head_size, head_size]. Must be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Starts from zero state whenNone. Whenstate_indicesis given (SM8x/SM90/SM100/SM103/SM120), this is instead the state pool[N_pool, num_sab_heads, head_size, head_size]and sequenceireads its initial state from rowstate_indices[i]; on SM100/SM103 every rank-4 positive, non-overlapping stride layout is accepted.output_final_state (bool) – Whether to output the final state. Default:
False.cu_seqlens (torch.Tensor) – Cumulative sequence lengths of shape
[num_seqs + 1], integer int32 or int64 dtype on the same CUDA device asq. Required for variable-length sequences (varlen mode); must not beNone. Repeated adjacent offsets represent legal zero-length sequences.use_qk_l2norm_in_kernel (bool) – Whether to L2-normalize each Q/K head with epsilon
1e-6. Normalization accumulates in float32 and rounds back to the input dtype before the chunked kernel. Default:False.output (torch.Tensor, optional) – Pre-allocated output tensor of shape
[total_seq_len, num_o_heads, head_size]wherenum_o_heads = max(num_q_heads, num_v_heads). Allocated automatically whenNone.output_state (torch.Tensor, optional) – Pre-allocated output state tensor. Packed, sequence-ordered shape
[num_seqs, num_sab_heads, head_size, head_size]. May be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Required whenoutput_final_state=True. Whenstate_indicesis given it is instead the output state pool[N_pool, ...]and sequencei’s final state is written to rowstate_indices[i], or to rowoutput_state_indices[i]when that is given (in place whenoutput_state is initial_state); it must be provided by the caller (auto-allocation is rejected, since a compact[num_seqs, ...]buffer would be indexed out of bounds by the pool slot ids). SM100/SM103 accepts the same positive, non-overlapping rank-4 stride layouts asinitial_state.state_checkpoints (torch.Tensor, optional) – Pre-allocated checkpoint tensor of shape
[total_checkpoints, num_sab_heads, head_size, head_size]. May be float32, bfloat16, float16, float8_e4m3fn, or float8_e5m2. Required whencheckpoint_every_n_tokens > 0. Context-parallel checkpointing is currently supported on SM8x, SM90, SM100, and SM120.checkpoint_cu_starts (torch.Tensor, optional) – Cumulative checkpoint counts of shape
[num_seqs + 1], int32 or int64 on the same CUDA device asq.checkpoint_cu_starts[i+1] - checkpoint_cu_starts[i]is the number of checkpoints for sequencei:seq_len_i // checkpoint_every_n_tokens, or(prefix_lens[i] + seq_len_i) // N - prefix_lens[i] // Nwithprefix_lens(N = checkpoint_every_n_tokens). Withcheckpoint_state_indicesonlycheckpoint_cu_starts[i]is read and[num_seqs]entries are also accepted. Required whencheckpoint_every_n_tokens > 0. Withoutcheckpoint_state_indicesthe values must be monotonic and consistent withcu_seqlens; this caller precondition is not checked at launch to avoid a device-to-host synchronization.checkpoint_every_n_tokens (int) – Store intermediate state every N tokens. Must be a multiple of the chunk size (64).
0disables checkpointing (default).use_cp (Literal["auto"] | bool, optional:) – Whether to use the architecture-specific context-parallel DSL implementation on SM8x, SM90, SM100/SM103/SM107 or SM120, where it is supported and low-parallelism heuristics match. SM100/SM103/SM107 uses the generated GDN CP-only four-stage implementation for structurally supported shapes; other legal configurations retain the CuTe-DSL implementation.
"auto"enables conservative routing,Truerequires CP support, andFalsedisables CP. Default:"auto".state_indices (torch.Tensor, optional) –
Int32 or int64 tensor of shape
[num_seqs](SM8x/SM90/SM100/SM103/SM120). When provided,initial_stateandoutput_stateare treated as a state pool whose first dimension is indexed by these slot ids rather than laid out in sequence order: sequenceireads its initial state from rowstate_indices[i]and writes its final state back to the same row (in place whenoutput_state is initial_state), or to rowoutput_state_indices[i]when that is given. This lets callers that keep a paged/indexed state pool avoid gathering the active rows into a packed buffer and scattering the result back. The pool may be any positive, non-overlapping rank-4 stride layout on SM100/SM103.None(default) keeps sequence-ordered row mapping without requiring the physical view itself to be contiguous.Without
output_state_indicesthe ids must be unique: as with any indexed scatter, two sequences sharing a slot id would concurrently write the same pool row across work tiles, leaving that row’s final state nondeterministic. Uniqueness is a caller precondition (not checked at launch, to avoid a per-call host sync); the caller’s slot allocator is expected to guarantee it. Withoutput_state_indicesread ids may repeat or be-1; only the rows a sequence writes must not be read or written by another sequence.backend="cudnn"requires frontend 1.31+, int32 indices and float32/bfloat16 pools with dense[H, V, K]rows and a 16-byte aligned slot stride. It supports identical input/output pool views or disjoint memory, and does not support checkpoints._cp_chunk_len (int, optional) – Internal context-parallel chunk-length override used for testing and tuning.
Nonelets the CP backend select the length automatically; an explicit value must be a multiple of 64.backend ({"auto", "flashinfer", "cake_gdn", "cudnn", "sm120_cute"}) –
autouses the native SM90/SM100/SM120 kernels and context-parallel routing, and may select cuDNN for measured SM100 raw-gate inference workloads. Automatic cuDNN selection requires both gate-fusion flags, BF16 inputs/gates, caller-normalized Q/K, separate compact float32 input/output states withoutput_final_state=True, and a caller-owned output. Explicit CP, pools and checkpoints retain native routing. Requires cuDNN frontend 1.31 GA and CuTe DSL 4.7 or newer. Cake kernels require an explicitcake_gdnrequest. Usebackend="cake_gdn", use_cp=Truefor Cake CP on SM100/SM103/SM107;use_cp=Falseretains Cake non-CP, and"auto"enters Cake CP only inside the measured SM100/SM103 band (BF16/FP16 I/O, FP32 initial and final state, gates, no checkpoints or state indices,4 * num_seqs * max(Hq, Hv) < SMsandmax_seqlen >= 4096; without amax_seqlenhint the band estimates the longest sequence asceil(total_seq_len / num_seqs)) or the SM107 bands (FP16 I/O; or BF16 I/O for one sequence of at most 65536 tokens, a multiple of 128, with int32cu_seqlens, FP32 initial and final state, gates, caller-normalized Q/K and the measured head geometries, where the non-CP kernel fails closed). Explicit Cake requests fail for unsupported inputs without falling back to another backend. The Cake non-CP route performs no device-to-host synchronization and no eager warm-up call before CUDA Graph capture (the one-time kernel compile/load per variant is the only eager step); seecu_seqlens_cpu. The explicitcake_gdnnon-CP route serveshead_size == 128with BF16 or FP16 I/O (one dtype for Q/K/V/output, caller-normalized Q/K, FP32 gates). State dtypes: FP32 with either I/O dtype; BF16, FP16 and FP8 (e4m3/e5m2) with BF16 I/O, the low-precision three only with an initial and a final state on the DV-split schedule. The initial state may be zero (None) or given and the final state is optional;state_indicespools need both. Checkpoints are limited to the frozen FP16-I/O FP32-state packed contract without an initial state and to the SGLang TP4 (H=4, HV=8) BF16-I/O BF16-state indexed DV-split contract at 64-token intervals. State rows must be dense[H, 128, 128](strides(stride(0), 16384, 128, 1)); the data pointers ofinitial_state,output_stateandstate_checkpointsand the slot stridestride(0)ofinitial_stateandoutput_stateneed only be element-aligned, not 16-byte aligned (sliced, offset or padded state pools give the same bits as 16-byte-aligned buffers): 16-byte-aligned state runs the unguarded body, element-aligned state a bit-identical guarded twin. The frozen(HEAD_GROUP_LOG2, IS_GQA, NUM_O_HEADS_LOG2)geometry set of each contract is enumerated byflashinfer.jit.cake_gdn.cake_gdn_prefill_geometries()fromcsrc/gdn/cake/manifest.json; on SM107 a DV-split key whose geometry only the full-DV set lists runs the full-DV record, and an unlisted geometry, schedule or architecture raisesCakeGDNUnsupportedError. The BF16-I/O FP32-state initial+final contract covers the Qwen3-Next / Qwen3.5 per-rank geometries for TP1, TP2, TP4 and TP8 in both physical schedules (DV-split and full-DV).cudnnruns cuDNN’s fused SM100 linear-attention engine throughflashinfer.cudnn.cudnn_chunk_gated_delta_rule().max_seqlen (int, optional) – Safe upper bound on the maximum logical sequence length, no larger than
total_seq_len. CP kernels use this host-side hint to bound their per-sequence launch grids without readingcu_seqlensback from the GPU. When omitted, CP usestotal_seq_len, which is correct for any batch; passing the exact maximum of a batched call lets CP launch smaller grids. The SM100/SM103use_cp="auto"band reads the hint (or the exact maximum fromcu_seqlens_cpu) as the longest sequence and, without either, the balanced-batch estimateceil(total_seq_len / num_seqs); pass the exact maximum so the band decision matches the Cake dispatcher’s.backend="cake_gdn"without CP uses the same bound (or the exact maximum fromcu_seqlens_cpu) to select its kernel variant; a loose bound is always correct and only affects whether the single-chunk kernel is eligible. The persistent grid does not depend on the bound: it follows the per-sequence lengths ofcu_seqlens_cpuwhen given (see below), else the residency rule.gate_domain ({"linear", "log"}) – Domain of precomputed
g: linear alpha (default) or natural-log alpha.Nonestill means no decay. Ignored when use_gate_in_kernel is set, since that flag makesga raw gate input.use_gate_in_kernel (bool) – Interpret
gas raw logits and computelog(alpha) = -exp(A_log) * softplus(g + dt_bias). cuDNN fuses this transform; other backends materialize the same FP32 alpha.A_log (torch.Tensor, optional) – Per-head
[num_sab_heads]parameters, required with use_gate_in_kernel. Float32, float16 or bfloat16, on q’s device.dt_bias (torch.Tensor, optional) – Per-head
[num_sab_heads]parameters, required with use_gate_in_kernel. Float32, float16 or bfloat16, on q’s device.beta_is_logit (bool) – Apply sigmoid to beta in FP32, round to beta’s input dtype, then accumulate in FP32. cuDNN fuses the transform; other backends materialize it. The default accepts precomputed beta.
cu_seqlens_cpu (Sequence[int] or torch.Tensor, optional) – Host copy of
cu_seqlens: a one-dimensional int32/int64 CPU tensor or a sequence of Python ints of lengthnum_seqs + 1holding the same values (vLLM’squery_start_loc_cpu, the prefix sums of SGLang’sextend_seq_lens_cpu). Used bybackend="cake_gdn"without CP, and on SM8x by theuse_cp="auto"and V32 rules, which need the exact longest sequence and keep the fused path without it. It is also required by the explicit experimentalsm120_cuteroute. When given, the Cake route validates it on the host (non-decreasing, first entry0, last entrytotal_seq_len; a mismatch raises before any launch), takes the exact longest sequence as the bound for variant selection, checks the checkpoint count exactly, and enables the ragged-batch launch grid: from the per-sequence lengths the host models the persistent kernel’s strided tile assignment against one CTA per tile and launches one CTA per tile when the modelled critical path shrinks by at least 4 % (same kernel binary, bitwise identical outputs); equal-length batches keep the residency grid. When omitted the route still reads nothing back from the device: the bound ismax_seqlen(ortotal_seq_len), the persistent grid follows the residency rule alone andcu_seqlensis resolved by the kernel at every launch, which clamps every offset into[0, total_seq_len]and treats a non-monotonic pair as an empty sequence – a documented no-op, never an out-of-bounds access (checkpoint_cu_startsconsistency stays a caller precondition). Int32cu_seqlensreaches the kernel as is; int64 is cast on the stream at every call, except on records that read int64 metadata directly (every Cake non-CP prefill record on CC 10.7), where int64 reaches the kernel as is too. Nothing is cached by tensor identity, so a buffer rewritten in place is always read live, including on CUDA Graph replay.output_state_indices (torch.Tensor, optional) – Contiguous int32 or int64 tensor of shape
[num_seqs]on the device ofq. Requiresstate_indices, an explicitoutput_statepool andoutput_final_state=True. Sequenceithen reads its initial state from rowstate_indices[i]ofinitial_stateand writes its final state to rowoutput_state_indices[i]ofoutput_state, which may be the same pool. A read id of-1starts the sequence from the zero state; a write id of-1skips its final-state write. A zero-length sequence writes nothing: its write row keeps its contents. Rows written by one sequence must not be read or written by another sequence of the same call.checkpoint_state_indices (torch.Tensor, optional) – Contiguous int32 or int64 destination rows for checkpoints, on the device of
q. Requirescheckpoint_every_n_tokens > 0.state_checkpointsis then a state pool[N_pool, num_sab_heads, head_size, head_size]whose rows are dense[H, V, K]blocks (the row stride may be padded), so a pool it shares withinitial_state/output_stateneeds that layout too. Thej-th checkpoint of sequenceiis written to rowcheckpoint_state_indices[checkpoint_cu_starts[i] + j]; a row of-1skips that checkpoint.checkpoint_cu_starts([num_seqs]or[num_seqs + 1]entries) only gives each sequence’s first entry, so a paged caller can pass its page table directly (seeflashinfer.gdn_paged.chunk_gated_delta_rule_paged()).prefix_lens (torch.Tensor, optional) –
Contiguous int32 or int64 tensor of shape
[num_seqs]on the device ofq: the number of tokens of each sequence processed before this call. Requirescheckpoint_every_n_tokens > 0. Checkpoints are then taken at absolute token positions: sequenceistores a checkpoint after itsn-th new token wheneverprefix_lens[i] + nis a multiple ofcheckpoint_every_n_tokens(1 <= n <= seq_len_i). Prefix lengths need not be multiples of the 64-token chunk; a sequence whose prefix is not chunk-aligned is chunked from its aligned absolute position, so every checkpoint falls on a chunk boundary. Sequenceithen has(prefix_lens[i] + seq_len_i) // N - prefix_lens[i] // Ncheckpoints (N = checkpoint_every_n_tokens); sizecheckpoint_cu_startsaccordingly.output_state_indices,checkpoint_state_indicesandprefix_lensare served by the native non-CP kernels on SM8x, SM90, SM100/SM103/SM107 and SM120 (backend="auto"or"flashinfer");use_cp=True,backend="cake_gdn"andbackend="cudnn"reject them.
- Returns:
When
output_final_state=False, the output tensor of shape[total_seq_len, num_o_heads, head_size]. Otherwise a tuple(output, final_state)wherefinal_statehas shape[num_seqs, num_sab_heads, head_size, head_size]— or, whenstate_indicesis given, the state pool[N_pool, ...]itself (i.e.output_state), whose rows named bystate_indices(byoutput_state_indiceswhen given;-1rows are not written) now hold the updated final states.- Return type:
torch.Tensor or Tuple[torch.Tensor, torch.Tensor]
Notes
backend="sm120_cute"explicitly selects the experimental RTX 6000D SM120 D=128 CuTe implementation. Fast calls require native finite FP32 log gates (gate_domain="log"), FP32 beta/state and matching immutablecu_seqlens_cpumetadata. Unsupported calls, including checkpoints, state pools, linear/raw gates and low-precision state, use FlashInfer’s official implementation. Stable automatic dispatch never selects it.Supports GQA (
num_q_heads > num_k_heads = num_v_heads) and GVA (num_v_heads > num_q_heads = num_k_heads).The final state layout is
[N, H, V, K].Requires SM8x (Ampere), SM90 (Hopper), SM100/SM103 (Blackwell), SM107 or SM120. The SM100 path requires
head_size == 128. On SM100/SM103,gdn_cpsupports structurally legal equal-head, GQA, and GVA shapes on CUDA 12.8, CUDA 12.9, and CUDA 13; SM107 supports the same shapes with annvccthat can emitcompute_107a. Other SM100 CP DSL routes require CUDA 13 andnvidia-cutlass-dsl[cu13]>=4.4.2(pip install flashinfer-python[cu13]).On SM8x the state may not be FP8.
use_cp="auto"selects CP on compute capability 8.0 whencu_seqlens_cpugives a longest sequence of at least 8192 tokens andnum_seqs * max(num_q_heads, num_v_heads)is at most 16.