flashinfer.fused_moe.QuantConfig

class flashinfer.fused_moe.QuantConfig(weight: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, activation: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, output: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, *, swizzled_scale_factors: bool | None = None, per_token_scale: bool | None = None, nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>)

Quantization scheme: MMA weight/activation formats plus the result format.

QuantConfig only checks that each axis is a QuantFormat. Legal combinations are runner capabilities (supported_quant_variants and supported_output_formats), not an allow-list in this dataclass.

Parameters:

  • weight (QuantFormat) – MMA weight and activation formats. An omitted axis is BF16, i.e. unquantized: QuantConfig() is BF16×BF16 and QuantConfig(weight=MXFP4) is MXFP4 weights with BF16 activations (W4A16). MXFP4×MXFP8 must be spelled with both axes.

  • activation (QuantFormat) – MMA weight and activation formats. An omitted axis is BF16, i.e. unquantized: QuantConfig() is BF16×BF16 and QuantConfig(weight=MXFP4) is MXFP4 weights with BF16 activations (W4A16). MXFP4×MXFP8 must be spelled with both axes.

  • output (QuantFormat) – Layer output format. Default BF16. Pass QuantFormat.FP16, not torch.float16.

  • swizzled_scale_factors (bool or None) – Whether block scale factors use the swizzled (vs linear) layout. None → backend default. Mirrors core’s swizzled_input_sf. Finer SfLayout (128x4 / 8x4 / linear) selection is deferred (design doc C42): unlike RoutingMethodType / ActivationType, SfLayout has no eval-safe __repr__, so exposing it here would break the eval(repr(cfg)) round-trip — a bool keeps this config serializable.

  • per_token_scale (bool or None) – Whether activations carry a per-token scale (vs per-tensor / block). None → backend default.

  • nvfp4_4over6 (NVFP44Over6Config or None) –

    NVFP4 “4over6” scale-candidate search for the activation quantization the kernels perform at runtime. Weights are quantized by the caller ahead of the call and are not affected.

    • unset (the default): read the legacy FLASHINFER_NVFP4_4OVER6* environment variables on every call (byte-for-byte the old behaviour).

    • None: 4over6 off. The environment is ignored.

    • NVFP44Over6Config(...): on with exactly that recipe. The environment is ignored, with no per-field merge.

    Only NVFP4 activations accept an explicit value (__post_init__ raises otherwise). A runner that has not threaded the setting into its quantizer rejects an explicit value instead of dropping it; see MoERunner.supports_nvfp4_4over6. A pinned recipe also pins the GEMM2-input scale of the weight pack: build it with CuteDslConfig.prepare_weights(..., nvfp4_4over6=<the same value>) or the call raises ValueError. An unset field is not emitted by __repr__, so the eval(repr(cfg)) round-trip still holds.

Raises:

  • ValueError – If nvfp4_4over6 is set (to None or a config) when activation is not QuantFormat.NVFP4.

  • TypeError – If nvfp4_4over6 is neither None nor an NVFP44Over6Config.

__init__(weight: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, activation: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, output: ~flashinfer.fused_moe.api.QuantFormat = QuantFormat.BF16, *, swizzled_scale_factors: bool | None = None, per_token_scale: bool | None = None, nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>) → None

Methods

__init__([weight, activation, output, ...])

Attributes

activation

nvfp4_4over6

output

pair

MMA (weight, activation) pair used for runner matching.

per_token_scale

swizzled_scale_factors

weight