flashinfer.fused_moe.CudnnGroupedGemmFp8PerTensorConfig

class flashinfer.fused_moe.CudnnGroupedGemmFp8PerTensorConfig

cuDNN grouped-GEMM FP8 per-tensor backend (grouped_mm_fp8).

  • Pipeline: expert-sorted permutation, GEMM1, the typed activation, GEMM2 and the finalize, on flashinfer.grouped_mm.grouped_mm_fp8().

  • Activations: the canonical per-tensor FP8 pack, E4M3 quantized with the static hidden_states_scale_global multiplier and hidden_states_scale=None.

  • Weights: prepare_weights() takes both static multipliers and folds them into the view.

  • The BF16 intermediate is requantized to E4M3 with the static intermediate_scale_global multiplier before GEMM2.

  • hidden_size must be a multiple of 16 and intermediate_size a multiple of 8.

  • Permute and finalize: FlashInfer’s moe_sort, moe_permute and moe_unpermute kernels.

  • Activation functions: SwiGLU with default scalars, GeGLU, GELU, ReLU, SiLU and Identity, on FlashInfer’s moe_activation kernel.

  • MoEFinalizeConfig(do_finalize=False) returns the unfinalized [gemm2_out, expert_weights, token_to_row].

  • Expert parallelism: assignments to non-local experts are masked out.

  • Routing weights: packed, or unpacked FP32 / BF16.

  • Each token’s topk_ids must name distinct experts.

  • Not in the default backend list.

__init__() → None

Methods

__init__()

prepare_activations(hidden_states_bf16, *, ...)

Quantize BF16 activations with one calibrated E4M3 multiplier.

prepare_weights(w1_bf16, w2_bf16, *, ...[, ...])

Quantize canonical BF16 weights to per-expert E4M3 with the static multipliers folded in.

supported(arch)