flashinfer.cudnn¶
cuDNN-backed attention kernels. These wrappers call into NVIDIA’s cuDNN runtime
for batch prefill and batch decode, and are reachable as backend="cudnn" on
BatchPrefillWithPagedKVCacheWrapper / BatchDecodeWithPagedKVCacheWrapper
(or directly) when cuDNN is available on the host GPU. For decode the wrapper
backend covers fp16/bf16 GQA with return_lse, CUDA graphs, multi-token
decode (q_len_per_req > 1, bottom-right causal), a left sliding window
(window_left) and attention sinks; it does not support RoPE, soft-cap
or fp8/NVFP4 KV. A sink at q_len_per_req == 1 is served when the cuDNN
stack’s SDPA engines accept it (cudnn-frontend 1.30+, whose default SM100 engine
takes the graph the backend engine declines); older frontends raise a
not-supported error at the first run.
cuDNN’s CuTe-DSL (“FROST”) SDPA engine for SM100 is a default engine of
cudnn-frontend 1.30+: it serves multi-token decode rows with a decode tile, and
an attention sink at q_len_per_req == 1 falls through to it when the backend
engine declines. No environment variable is involved. With that frontend
installed, the decode wrapper’s backend="auto" resolves to cudnn on
SM100 / SM103 for the multi-token rows (2 <= q_len_per_req <= 4) of fp16/bf16
head_dim-128 GQA models when that tile has at least 32 packed rows per CTA and 64
CTAs, where it measures at 0.35-0.95x fa2 (multi-token rows without tensor cores,
which have no fa2 kernel, take cuDNN whenever its decode path can run them), and
for single-token decode of fp16/bf16 head_dim-256 GQA models (groups 4, 8 and 16)
with 96 to 256 (batch x KV heads) CTAs, where the d256 decode tile measures at
0.5-0.95x fa2; FLASHINFER_DECODE_AUTO_CUDNN overrides the choice. Under CUDA graphs auto
takes cuDNN only with a caller-owned block_tables (the auto-built table cannot
grow once captured), and the resolution is frozen after the first plan. The
performance-driven choice also requires that the frontend’s FROST engines can run
on the install: its own CuTe-DSL check (nvidia-cutlass-dsl 4.7.0 or newer,
satisfied by FlashInfer’s package minimum of 4.8.0) must pass, otherwise the
backend engine would serve the graph and fa2 is kept. Callers passing
fixed_split_size or disable_split_kv (fa2’s batch-invariance controls)
keep fa2 wherever fa2 can serve the rows. A CUDA-graph wrapper also tells
cudnn-frontend 1.31+ that the graph is replayed
(pygraph(is_cuda_graph_replay_expected=True)), so the d256 decode tile leads
with its split-KV plan, and the single-token d256 band then starts at 64 CTAs.
Compatible decode runs and replans retain the prepared cuDNN graph. Planning
still stages changing KV lengths and, unless the caller supplies a dense GPU
block_tables, constructs that table from CSR metadata. CUDA Graph replay
does not include this host planning work. fast_decode_plan uses the regular
cuDNN planner for these updates; its FA2/FA3 copy-elision does not apply to
cuDNN. workspace_size currently raises for both explicit and auto-selected
cuDNN, rather than returning another backend’s workspace requirements.
FP16/BF16 graph execution can write base-2 LSE directly, avoiding a separate
conversion. This requires cuDNN 9.27+ headers and runtime, an FE exposing the
capability, and backend support for the requested graph. Older or unsupported
configurations retain natural-log Stats plus conversion; FP8 handling is
unchanged. Prefill callers requesting lse_base="ln" still receive the
natural-log output directly. Each requested LSE base/layout is a graph
specialization: warm it before CUDA graph capture.
|
Batched decode attention with paged KV cache, backed by cuDNN SDPA. |
|
Batched prefill attention with paged KV cache, backed by cuDNN SDPA. |
Ragged prefill: cached geometry and warmup¶
BatchPrefillWithRaggedKVCacheWrapper separates sequence metadata planning
from repeated attention execution. With shape overrides, cuDNN graphs are
cached by capacity class, while device indptrs carry the live lengths. Hq/Hkv,
head dimensions, dtypes, tensor strides, attention scale, mask and requested
Stats layout remain part of the graph signature; they are not padded into a
larger head-count class.
On SM100/SM107, qualified BF16 expanded MLA (D192/V128, equal Q/KV heads)
and FP16/BF16 D128/V128 ragged prefill with cuDNN Frontend 1.31 or newer and its matching
native extension use bounded batch/Q/KV classes for short-Q, long-KV prefixes.
The qualified range is B1–4, 4–64 query heads, Q2–1024, and KV2K–32K with
KV at least four times Q; D128 supports integral GQA groups 1/2/4/8/16.
Q2–128 share the Q128 class, while Q=1 keeps the separate decode class.
Batch and lengths round up to powers of two within the qualified range;
Q129 and Q241 can share the same Q256 graph. The graph also receives a packed
Q capacity, allowing either cuDNN provider to reserve workspace and choose
parallelism from useful bounds. A bounded declaration does not force an engine
or split count. In CUDA Graph mode, the wrapper’s initialized total-query
capacity also bounds packed workspace and is part of the graph cache key.
Replanning fewer live tokens preserves that lifetime capacity, so older captures
remain valid. No device length read or extra preparation is added to run().
Wrappers without an initialized lifetime capacity retain the batch/Q envelope.
Other inputs retain their existing cache policy.
Paged backend="auto" additionally considers cuDNN on SM100 for causal
BF16 HND D128/V128 prefixes with page size 16, B1–4, 8–64 query heads,
GQA1/2/4/8/16, Q64–1024, and KV2K–32K at least four times Q. Small workloads
with total_query_tokens * Hq * max_KV < 8 * 1024 * 1024 retain FA2 to
avoid paged setup and split/combine overhead. This admission
requires stable cuDNN Frontend 1.32 or newer, its matching native paged
capability, a compatible FROST runtime, and cuDNN 9.24 or newer. Bounded
paged plans use explicit per-request Q/KV lengths with token-unit physical
origins; mixed cumulative-Q/per-request-KV lengths and FP8 retain their
cuDNN 9.25 floor. Older stacks
and unsupported attention options retain their existing auto route.
The wrapper reuses bounded plans and refreshes metadata in plan();
first use of a new bucket still requires prewarming. GPU replay gains do not
imply lower eager submission cost or higher serving throughput.
Explicit paged backend="cudnn" also reuses prepared plans on cuDNN 9.24
when both per-request Q and KV lengths are available. This does not require
bounded auto dispatch. The single-token GQA Stats workaround retains its
cuDNN 9.27 guard.
For FP16/BF16 cuDNN prefill, runtime q_scale and k_scale multiply the
attention score scale, and v_scale multiplies the output without changing
LSE. Python scalars are fixed within a capture; scalar float32 CUDA tensors
are read again on replay. Tensor Q/K scales use cuDNN’s tensor attention-scale
contract, so the frontend may choose a different engine than for a constant
scale. Calls without calibration scales retain the prepared execution path.
A new graph signature can require graph construction and kernel compilation.
Warm the signatures needed by the serving schedule before CUDA Graph
capture or latency-sensitive requests, using the existing plan and run
methods. No separate preparation API is required. In particular:
Warm each batch/Q/KV capacity class that the schedule will use, with the same strides, scale, causal flag and
return_lse/lse_layoutas execution.The existing
max_token_per_sequenceandmax_sequence_kvplan arguments can declare known upper bounds. For example, use Q256/KV4096 for both a Q129/KV2049 warmup and a Q241/KV3001 request. Device indptrs still supply the true lengths; the buffers do not need dummy tokens up to those bounds.Allocate caller workspace and output/Stats buffers before capture. Use a side stream for warmup, establish stream dependencies, and then capture
runwith those buffers. Replays use the captured plan and addresses.Re-plan metadata when the serving batch changes. CPU indptrs, or CPU mirrors where supported, keep shape discovery off the GPU critical path. Repeated
runcalls with the same plan only bind current tensors and launch.An unprepared class, a changed layout/dtype/mask, or an evicted graph-cache entry can need fresh preparation. Do not treat the initial warmup as a promise that arbitrary future signatures will be compile-free.
A smaller declaration adds cache classes compared with the broad fallback. Measure startup cost and steady-state GPU replay separately; per-step host planning and eager submission are different measurements again. In particular, a long declared sequence bound or a mostly empty batch can make the selected parallelism conservative. Capacity is not a measurement of active work.
Linear attention¶
cuDNN’s fused SM100 linear-attention engines, reachable either directly or as
backend="cudnn" on flashinfer.chunk_gated_delta_rule(),
flashinfer.chunk_gated_delta_rule2(),
flashinfer.chunk_gated_delta_product() and
flashinfer.recurrent_kda(). "cudnn" is never selected implicitly for
GDN or KDA, both of which have FlashInfer kernels of their own; GDN-2 and GDP
have none, so their "auto" resolves here.
These wrappers gate on one thing only: cudnn-frontend 1.29+ with the
cutedsl extra, the release whose graph.gdn / graph.gdn2 /
graph.gdp / graph.kda nodes take gate_domain. There is no cuDNN backend-version floor
– the FROST engines behind those nodes are CuTeDSL kernels the frontend
compiles itself. Every other requirement, including the SM100 family
(SM100-SM103 and SM107), the head dims, the input dtypes and the head-count
relations, belongs to the engine, which declines a graph it cannot serve (the
per-engine reason is logged by the frontend; the raised
cudnnGraphNotSupportedError itself is generic). Arguments
FlashInfer has that cuDNN’s entry points do not – state checkpointing, indexed
state pools, the context-parallel delta rule, speculative decode – are
rejected by the routing layer before the call.
The recurrent state crosses this boundary untransposed. FlashInfer holds it
V-major as [N, H, V, K] and so does cuDNN, so initial_state and
output_state buffers are passed straight through. cuDNN’s ops take the
state in float32 or bfloat16 and return final_state in whichever was
given, so a bfloat16 state pool crosses with no copy at all. The GDN and GDP
forget gates are linear-space alpha at this boundary and cross as
gate_domain="linear"; the GDN-2 and KDA gates are log-space on both sides.
|
Chunked Gated DeltaProduct prefill on cuDNN's fused SM100 engine. |
|
Chunked Gated Delta Rule prefill on cuDNN's fused SM100 engine. |
|
Chunked Gated Delta Rule 2 prefill on cuDNN's fused SM100 engine. |
|
Kimi Delta Attention prefill on cuDNN's fused SM100 engine. |