flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_nvfp4¶
- flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_nvfp4(q: Tensor, k: Tensor, v: Tensor, cu_seqlens: Tensor, out: Tensor | None = None, *, cu_seqlens_host: Sequence[int] | None = None, softmax_scale: float | None = None, pv_mode: str = 'fp4') Tensor¶
Experimental NVFP4 non-causal packed-varlen self-attention for MiniMax-H3 on SM120 (GB202), following the SageAttention3 FP4 recipe.
Computes, for every segment
[a, b)ofcu_seqlensand every headh:out[a:b, h] = softmax(q[a:b, h] @ k[a:b, h]^T * softmax_scale) @ v[a:b, h]
with block-scaled NVFP4 tensor-core operands for both the scores and the value product and an FP32 softmax, as in SageAttention3 (
sageattention3_blackwell): K rows are centred by the segment’s per-channel mean and Q rows by their 128-row block’s mean (the block-mean termqm K^Tis added back inside the kernel, so the softmax is unchanged), Q / K / V are stored as E2M1 codes with one UE4M3 scale per 16 elements (block amax / 6; V transposed with per-16-key scales), QK^T and PV both runmma.sync.m16n8k64 kind::mxf4nvf4 block_scale scale_vec::4X, and the probabilities use Sage3’s two-level scheme (row maximum brought to 448 * 6, one UE4M3 scale per 16 keys, E2M1 codes); the output is rounded to BF16 once. Runtime-variable generated programs serve anytokensand any segment lengths (including empty segments and lengths below one tile), two or three programmatic-dependent launches per call on the current stream (minimax_h3_sm120_nvfp4_programs; gridsminimax_h3_sm120_nvfp4_launch_grids): a plan of 2048 to 16384 padded tokens runs the fused pre-processing (K sums and Q block means, a grid barrier, then the quantized operands, in one cooperative launch); any other plan runs the statistics launch, which also quantizes Q and the block-mean rows, then the K / V quantizer. On a part of 170 or 188 SMs every plan of at least 2048 padded tokens runs the overlapped form instead (minimax_h3_sm120_nvfp4_overlap_grids): head-major pre-processing producers start first on a few SMs (a plain launch), the attention launch, a programmatic dependent of the producers, starts on the remaining SMs, each head’s attention waiting for its operands, and a second attention launch, a programmatic dependent of the first, takes the remaining work on the producers’ SMs as they drain; the outputs are bitwise the same. The attention launch takes the warp-specialised form on parts of at least 188 SMs when the mean segment (empty segments included) is at least 1536 and below 16384 tokens. The quantized operands live in a grow-only workspace per (device, stream), and per capture for a call captured into a CUDA graph (workspace_bytes_nvfp4), whose buffer names are shared withpv_mode="fp8". The programs ship forsm_120aalone: a call with a non-empty segment on a device of another compute capability than 12.0 raisesRuntimeError.Concurrency: safe on any number of streams, host threads, CUDA graphs and MPS clients. Exec graphs instantiated more than once from one captured graph (or cloned as child graphs) share this op’s workspace and must not run concurrently.
CUDA graphs: the first call for a given
cu_seqlens_hostbuilds the plan’s device tables from host lists, a pageable host-to-device copy that a stream capture does not allow. Call the op once for that plan outside the capture before capturing it; later calls reuse the tables, and the capture then records kernel launches only. This holds withCUDA_MODULE_LOADING=LAZYas well: once the tables exist, the first launch of each program may happen inside the capture.The E2M1 scores and probabilities carry a larger quantization error than the FP8 operator (about 4x the relative L2 error on Gaussian inputs, the same as SageAttention3’s own); this route is an experimental precision/latency trade-off and is validated against the FP32 oracle with the FP4 block-scaled tolerance
atol = 1.0, rtol = 0.1(see the tests), not the FP8 operator’s0.1.- Parameters:
q (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).k (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).v (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).cu_seqlens (torch.Tensor) –
int32tensor of shape[segments + 1]on the same device withcu_seqlens[0] = 0, non-decreasing entries andcu_seqlens[-1] = tokens.out (Optional[torch.Tensor]) – Optional pre-allocated output of the same shape/dtype as
q; allocated when omitted.cu_seqlens_host (Optional[Sequence[int]]) – Host copy of
cu_seqlens(avoids a synchronizing device-to-host copy). The segment plan is cached per(cu_seqlens, heads, device).softmax_scale (Optional[float]) – Softmax scale; defaults to
1 / sqrt(128).pv_mode (str) –
"fp4"(default): the SageAttention3-recipe operator above, unchanged."fp8": the opt-in NVFP4-QK / FP8-PV route – the scores run on block-scaled NVFP4 operands (Q rotated by a signed Hadamard transform and K centred by the segment’s per-channel mean, one UE4M3 scale per 16 channels under a per-row / per-(segment, head) prescale) while the probabilities and V stay E4M3 as inminimax_h3_sm120_varlen_attention_fp8()(per-channel V scales, FA3 key permutation); three programmatic-dependent launches (fused statistics, quantizer, attention). It trades the FP4 probabilities for a smaller error (relative L2 about 0.14 on Gaussian inputs against about 0.19 for"fp4"and 0.05 for the FP8 operator) at a lower latency than the FP8 operator; see the tests. The same workspace names are reused; the arrival counters of the statistics launch are private to the (device, stream), and to the capture for a call captured into a CUDA graph. Any other value is aValueError.
- Returns:
bfloat16[tokens, heads, 128]attention output.- Return type:
torch.Tensor