flashinfer.mla.SparseMLASm120DecodeConfig

class flashinfer.mla.SparseMLASm120DecodeConfig(d_qk: int, page_block_size: int, max_num_tokens: int, topks: frozenset[int], min_topk: int, max_num_heads: int, kv_cache_format: str = 'fp8', bytes_per_token: int = 0, head_counts: frozenset[int] | None = None, topk_is_runtime: bool = True, extra_page_block_sizes: frozenset[int] = frozenset({}))

Instantiated decode-kernel set for one SM120 sparse-MLA kernel family.

Decode-form calls (num_tokens <= max_num_tokens) prefer a standalone decode kernel when their shape matches one of the instantiations described here. For FP8, the prefill orchestrator can serve remaining decode-form shapes in its own envelope; NVFP4 currently uses the same exact head/top-k envelope for both kernels. Crossover calibration may route an eligible shape to prefill. This config describes decode only.

d_qk

Query/key head dim served by this family (512 for DSv4 / GLM53_NOPE, 576 for DSv3.2 / GLM-NSA, 1088 for the DOTS3_SWA sliding-window family, whose d_v is then 1024).

Type:

int

page_block_size

The only KV page block size the decode kernels are instantiated for.

Type:

int

max_num_tokens

Largest num_tokens routed to the decode kernels (inclusive).

Type:

int

topks

The calibrated top-k values (the crossover sweep points). When topk_is_runtime is true, this documents measured values rather than the eligibility boundary; otherwise it is the exact set.

Type:

frozenset[int]

min_topk

Smallest legal topk (the indices-row width). 513 for the sliding-window family (the window must fit the buffer); 1 elsewhere.

Type:

int

max_num_heads

Upper bound of the head-count envelope.

Type:

int

kv_cache_format

Packed cache format described by this entry ("fp8" or "nvfp4").

Type:

str

bytes_per_token

Logical packed-cache bytes per token.

Type:

int

head_counts

Exact instantiated head counts when the kernel has no runtime-head fallback. None means every count in [1, max_num_heads].

Type:

Optional[frozenset[int]]

topk_is_runtime

Whether every topk >= min_topk is accepted. When false, only values in topks are instantiated.

Type:

bool

extra_page_block_sizes

Exact page sizes accepted by the optional secondary cache when the family has a finite set exposed here. Empty means unspecified.

Type:

frozenset[int]

__init__(d_qk: int, page_block_size: int, max_num_tokens: int, topks: frozenset[int], min_topk: int, max_num_heads: int, kv_cache_format: str = 'fp8', bytes_per_token: int = 0, head_counts: frozenset[int] | None = None, topk_is_runtime: bool = True, extra_page_block_sizes: frozenset[int] = frozenset({})) → None

Methods

__init__(d_qk, page_block_size, ...[, ...])
supported_num_heads()	Sorted instantiated head counts, including any runtime-H envelope.
supported_topk([num_heads])	Sorted calibrated top-k values for num_heads (or any head count).
supports_decode(num_heads, topk, *[, ...])	True iff a decode-form call with this shape is decode-instantiated.

Attributes

bytes_per_token
extra_page_block_sizes
head_counts
kv_cache_format
topk_is_runtime
d_qk
page_block_size
max_num_tokens
topks
min_topk
max_num_heads