flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_fp8¶
- flashinfer.diffusion_ops.minimax_h3_sm120_varlen_attention_fp8(q: Tensor, k: Tensor, v: Tensor, cu_seqlens: Tensor, out: Tensor | None = None, *, cu_seqlens_host: Sequence[int] | None = None, softmax_scale: float | None = None) Tensor¶
FP8 (E4M3) non-causal packed-varlen self-attention for MiniMax-H3 on SM120 (GB202).
Computes, for every segment
[a, b)ofcu_seqlensand every headh:out[a:b, h] = softmax(q[a:b, h] @ k[a:b, h]^T * softmax_scale) @ v[a:b, h]
with FP8 E4M3 tensor-core operands and an FP32 softmax: Q is quantized per token, K per 128-key block after subtracting its segment’s per-channel mean (an exact softmax shift), the probabilities are stored as E4M3 with a 2^8 exponent bias and V per (segment, channel). Both QK^T and PV run E4M3
mma.syncm16n8k32 MMAs in the block-scaledkind::mxf8f6f4form with unit scales (bitwise the dense form; GB202 GeForce boards issue the dense FP32-accumulating form at half rate); the output is rounded to BF16 once. Three runtime-variable generated programs, launched in order on the current stream (the K / V statistics with their per-segment fold, the quantizer, the attention), serve anytokensand any segment lengths (including empty segments and lengths below one tile); the grids areminimax_h3_sm120_quant_varlen_attention_launch_grids. A plan of 56 heads whose attention work (the sum of the squared segment lengths) lies in [2^23, 24576^2] runs, on a part of 170 or 188 SMs, the overlapped form instead (minimax_h3_sm120_quant_varlen_attention_programs,minimax_h3_sm120_quant_varlen_attention_overlap_grids): the head-major pre-processing producers start first and the attention starts each head as soon as its operands are written, with the same output bit for bit; above the band such a plan of at most 65536 (170 SMs) or 98304 (188 SMs) tokens takes its pre-processing from the head-major producers over every SM in place of the first two launches, with the same operands bit for bit. The quantized operands live in a grow-only workspace per (device, stream), and per capture for a call captured into a CUDA graph (workspace_bytes); the most recent 8 workspace sets and 256 segment plans are kept. The programs ship forsm_120aalone: a call with a non-empty segment on a device of another compute capability than 12.0 raisesRuntimeError.Concurrency: safe on any number of streams, host threads, CUDA graphs and MPS clients. Exec graphs instantiated more than once from one captured graph (or cloned as child graphs) share this op’s workspace and must not run concurrently.
- Parameters:
q (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).k (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).v (torch.Tensor) – Contiguous
bfloat16CUDA tensors of shape[tokens, heads, 128](packed THD layout; MiniMax-H3 uses 56 heads).cu_seqlens (torch.Tensor) –
int32tensor of shape[segments + 1]on the same device withcu_seqlens[0] = 0, non-decreasing entries andcu_seqlens[-1] = tokens.out (Optional[torch.Tensor]) – Optional pre-allocated output of the same shape/dtype as
q; allocated when omitted.cu_seqlens_host (Optional[Sequence[int]]) – Host copy of
cu_seqlens(avoids a synchronizing device-to-host copy). The segment plan is cached per(cu_seqlens, heads, device)(most recent 256).softmax_scale (Optional[float]) – Softmax scale; defaults to
1 / sqrt(128).
- Returns:
bfloat16[tokens, heads, 128]attention output.- Return type:
torch.Tensor