flashinfer.gdn_prefill¶
Gated Delta-Rule prefill-side kernels. chunk_gated_delta_rule is the
chunked GDN scan that initializes the recurrent state from a full prompt
before the decode loop takes over.
|
Chunked Gated Delta Rule (GDN) attention for prefill. |
Cake GDN non-CP backend: host-free launch¶
chunk_gated_delta_rule(..., backend="cake_gdn") with use_cp=False or
"auto" runs the Cake non-CP prefill kernels on SM100a/SM103a. The route
performs no device-to-host synchronization: the sequence count and the token
count are tensor shapes, the kernel variant and the persistent grid follow an
upper bound on the longest sequence, and the kernel resolves cu_seqlens
on the device at every launch (offsets are clamped into
[0, total_seq_len]; a non-monotonic pair is an empty sequence, a
documented no-op rather than an out-of-bounds access).
Pass the host copy of the offsets an engine already holds as
cu_seqlens_cpu (keyword-only; a CPU int32/int64 tensor or a sequence of
Python ints of length num_seqs + 1 – vLLM’s query_start_loc_cpu, the
prefix sums of SGLang’s extend_seq_lens_cpu). The route then validates
the offsets on the host before any launch and uses the exact longest sequence
to pick the kernel and grid. Without it, max_seqlen (or
total_seq_len) is the bound; a loose bound is always correct and only
affects the grid size and whether the stateless single-chunk kernel is
eligible. Other backends ignore cu_seqlens_cpu.
Int32 cu_seqlens / state_indices / checkpoint_cu_starts reach the
kernel as is; int64 metadata is cast on the stream at every call. Nothing is
cached by tensor identity, so a metadata buffer rewritten in place is always
read live. CUDA Graph capture needs no eager warm-up call for the same
metadata: the one-time nvcc compile and load of a kernel variant is the
only eager step, and replay reads the live buffers. The consistency of
checkpoint_cu_starts with cu_seqlens and the checkpoint interval
remains a caller precondition. See csrc/gdn/cake/README.md (“Non-CP
prefill: host-free launch”) for the full contract.
Cake GDN CP context-parallel backend¶
Enable Cake CP explicitly with
chunk_gated_delta_rule(..., backend="cake_gdn", use_cp=True).
backend="auto" and backend="flashinfer" keep the existing FlashInfer
CP routing. With backend="cake_gdn", use_cp=False or "auto"
continues to select the non-CP Cake backend. On SM107a, "auto" with BF16
inputs and FP32 initial and final state raises CakeGDNUnsupportedError
when the longest sequence has at least 8192 tokens and num_seqs * num_heads
is at most 32, or at least 4096 tokens and at most 8: the default backend is
faster there.
The Cake GDN CP backend covers the complete legal public context-parallel input
domain on SM100a/SM103a/SM107a. It preserves T precompute -> MN
precompute -> state fixup -> CP prefill and supports the public dtype,
head-mapping, optional gate/scale, int32/int64 cu-seqlens and state indices,
mixed or all-empty varlen batches, Q/K L2 normalization, checkpoints, output,
and arbitrary positive non-overlapping packed/indexed/in-place state strides.
On SM100a/SM103a/SM107a, the gdn_cp route uses checked-in CUDA sources for
FP16/BF16 inputs plus FP32/FP16/BF16 state and FP32 checkpoints. SM100a/SM103a
are supported with CUDA 12.8, CUDA 12.9, and CUDA 13; SM107a needs a toolkit
whose nvcc can emit compute_107a. Other SM100 context-parallel DSL
routes, including FP8 state or checkpoints, require CUDA 13 and
nvidia-cutlass-dsl[cu13]>=4.4.2.
Internally, the public dispatcher caches the shape-specific plan and workspace
but launches the native composite directly, so API-allocated outputs and
rotating input buffers can change address without rebuilding the plan. A
fixed-address internal prepared object can use CUDA Graph replay; indexed
in-place state always stays on the direct composite so preparation cannot
advance aliased recurrent state. Invalid inputs and unsupported architectures
fail closed; explicit Cake CP requests do not fall back to another
CP implementation. The existing chunk_gated_delta_rule API
remains the only public entry point. Every TMA-backed stage passes its
descriptor through CUDA’s __grid_constant__ kernel-argument ABI; the
backend does not retain a process-lifetime descriptor arena.