flashinfer.fused_moe.FrostLowLatencyConfig

class flashinfer.fused_moe.FrostLowLatencyConfig(max_num_tokens: int = 2)

Low-token routed MoE on SM80/SM90/SM100/SM103/SM107/SM120.

SM80 supports BF16/FP16 inputs and weights.

SM90 supports BF16, FP16, ordinary per-tensor FP8, DeepSeek FP8 and NVFP4/MXFP4/MXINT4-weight with BF16/FP16-activation FMA. Explicit opt-in; omitted from the default backend list. Autotuning selects Tensor Core tile sizes or a scalar FMA tactic (top-k <= 32) using the same prepared weights and numerical contract. Both are multi-kernel pipelines, with precomputed routing and no EP communication. The MX Tensor Core path overlaps its fused GU/activation/quantizer and Down GEMM through PDL. Set max_num_tokens up to 8 for tuning; the default admits at most 2. Prepare other backend views as well to permit fallback for larger inputs.

MXFP4/MXFP8 weights with MXFP8 inputs support the same ten typed activations as BF16, through TC or FMA. Non-gated weights omit gate rows. BF16 or FP16 weights/inputs use FMA with gated or non-gated activations and top-k at most 32. The activation is rounded to the input dtype and expert outputs to the configured BF16/FP16 output dtype. Combine uses ordered, separate FP32 multiplication and addition. Packed routing scores retain BF16 rounding regardless of the input/output dtype. This is distinct from the MX intermediate quantization/combine contract. SM80, SM90 and SM120 also offer grouped BF16/FP16 Tensor Core candidates at H=2048/I=512/top-k=10, T=4 or 8, with finalized matching output dtype. SM80 admits default SwiGLU and ReLU2; SM90/SM120 add GeGLU, and ReLU2/SiLU use T8 on SM90 and T4/T8 on SM120. SM120 also admits GeGLUTanh and default SiTU/SwiGLUStep at T4/T8 with matching BF16/FP16 types. SM80 additionally admits BF16 default SwiGLU at H=7168/I=2048/top-k=8 and T=4 or 8. These projections share the typed intermediate and ordered combine contract; FMA remains the untuned default and a tuning candidate.

Ordinary per-tensor E4M3 FP8 uses static calibration multipliers, with BF16 FC1 preactivation rounding as in the CUTLASS per-tensor backbone. Activation and intermediate quantization remain fused into FC1; FC2 produces BF16 expert/final outputs with ordered FP32 multiply/add combine. On SM90, finalized default SwiGLU/SiTU at H=2048/I=512/top-k=10, T=4 or 8, and ReLU2 at T=8 also offer grouped E4M3 WGMMA with bounded-K64 FP32 partial promotion. SM120 adds grouped E4M3 warp MMA for the same shape, with default SwiGLU/SiTU/ReLU2 at T4/T8. Static scales stay live, and FMA remains the untuned default and tuning candidate. The TRT-LLM per-tensor path retains FP32 preactivations, so its rounding contract differs even with identical quantized weights and scales.

E4M3 weight-only mode uses BF16/FP16 inputs and intermediates, with per-expert FP32 weight scales. FC1 preactivation rounds to the input dtype; expert outputs round to the independently configured BF16/FP16 dtype. No activation quantization is required.

DeepSeek FP8 preserves FP32 preactivation and 128x128 weight block scales. BF16 input is quantized per 128 values; prequantized E4M3 input retains its supplied FP32 scales. The intermediate is dynamically quantized to E4M3 before FC2; expert outputs and ordered combine follow BF16 rounding.

FP4 weight-only modes preserve canonical E2M1 payloads with BF16/FP16 input, preactivation and intermediate rounding. Expert outputs use the configured BF16/FP16 output dtype. NVFP4 uses linear E4M3 K16 scales; MXFP4 uses E8M0 K32 scales and FP32 dequantization to retain its wider range. It uses software decoding on SM90 and native FP4 conversion on Blackwell/Rubin, sharing the BF16 FMA schedule and typed activation/combine implementation. H must be divisible by 128; logical I is padded to 128 from multiples of 32.

MXINT4 weight-only mode preserves signed INT4 payloads and BF16 K32 scales. It retains FP32 preactivation, then BF16/FP16 intermediate/expert rounding and ordered FP32 combine through the same two FMA kernels.

NVFP4 weights/inputs use shared FMA with H divisible by 512, I by 128, and top-k at most 32. It retains BF16 activation/expert rounding and separate FP32 multiply/add combine; the 4over6 recipe is not supported.

With MXFP8 activations and MXFP8/MXFP4 weights, or NVFP4 activations and NVFP4 weights, expert IDs outside [0, num_experts) mark unrouted slots. For these format pairs, finalization ignores those slots, including their scores, and raw output writes zero rows with the usual identity route mapping. Raw routing scores are returned unchanged apart from packed BF16 rounding; a downstream reducer must also exclude unrouted slots before multiplying scores that may be NaN or Inf. Other format pairs require expert IDs in [0, num_experts).

__init__(max_num_tokens: int = 2) → None

Methods

__init__([max_num_tokens])
prepare_bf16_weights(w1, w2, *[, activation])	Prepare BF16 [up, gate] or non-gated weights, padding I to 128.
prepare_deepseek_fp8_weights(w1, w1_scale, ...)	Preserve E4M3 weights and FP32 128x128 block scales without requantizing.
prepare_fp16_weights(w1, w2, *[, activation])	Prepare FP16 [up, gate] or non-gated weights, padding I to 128.
prepare_fp8_per_tensor_activations(...)	Quantize BF16 activations with one calibrated E4M3 multiplier.
prepare_fp8_per_tensor_weights(w1_bf16, ...)	Prepare E4M3 weights with static input/intermediate calibration scales.
prepare_fp8_weight_only_weights(w1, ...[, ...])	Prepare E4M3 weights for BF16/FP16 inputs without requantizing the payload.
prepare_mxfp4_weight_only_weights(w1, ...[, ...])	Preserve CUDA E2M1 weights and linear E8M0 scales per 32 K values.
prepare_mxint4_weight_only_weights(w1, ...)	Preserve signed INT4 payloads and linear BF16 scales per 32 K values.
prepare_nvfp4_weight_only_weights(w1, ...[, ...])	Preserve canonical E2M1 weights and linear E4M3 K16 block scales.
prepare_nvfp4_weights(w1, w2, *[, activation])	Quantize BF16 source weights to the canonical CUTLASS NVFP4 view.
prepare_weights(w1, w1_scale, w2, w2_scale, *)	Pack canonical MX weights; pass activation for non-gated weights.
supported(arch)

Attributes

max_num_tokens