flashinfer.fused_moe.cutlass_fused_moe¶
- flashinfer.fused_moe.cutlass_fused_moe(input: ~torch.Tensor, token_selected_experts: ~torch.Tensor, token_final_scales: ~torch.Tensor, fc1_expert_weights: ~torch.Tensor, fc2_expert_weights: ~torch.Tensor, output_dtype: ~torch.dtype, quant_scales: ~typing.List[~torch.Tensor], fc1_expert_biases: ~torch.Tensor | None = None, fc2_expert_biases: ~torch.Tensor | None = None, input_sf: ~torch.Tensor | None = None, swiglu_alpha: ~torch.Tensor | None = None, swiglu_beta: ~torch.Tensor | None = None, swiglu_limit: ~torch.Tensor | None = None, tp_size: int = 1, tp_rank: int = 0, ep_size: int = 1, ep_rank: int = 0, cluster_size: int = 1, cluster_rank: int = 0, output: ~torch.Tensor | None = None, enable_alltoall: bool = False, use_deepseek_fp8_block_scale: bool = False, use_w4_group_scaling: bool = False, use_mxfp8_act_scaling: bool = False, min_latency_mode: bool = False, use_packed_weights: bool = False, use_wfp4afp8_humming: bool = False, tune_max_num_tokens: int = 8192, enable_pdl: bool | None = None, activation_type: ~flashinfer.tllm_enums.ActivationType = ActivationType.Swiglu, swizzled_input_sf: bool = True, use_fused_finalize: bool | None = None, profile_ids: ~typing.List[int] | None = None, workspace_buffer: ~torch.Tensor | None = None, *, clamped_relu2_limit: ~torch.Tensor | None = None, situ_beta: ~torch.Tensor | None = None, situ_linear_beta: ~torch.Tensor | None = None, backend: str = 'cutlass', nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>) Tensor¶
Compute a Mixture of Experts (MoE) layer using CUTLASS backend.
This function implements a fused MoE layer that combines expert selection, expert computation, and output combination into a single operation. It uses CUTLASS for efficient matrix multiplication and supports various data types and parallelism strategies.
- Parameters:
input (torch.Tensor) – Input tensor of shape [num_tokens, hidden_size]. Supports float, float16, bfloat16, float8_e4m3fn, and nvfp4. For FP8, the input must be quantized. For NVFP4, both quantized and non-quantized inputs are supported.
token_selected_experts (torch.Tensor) – Indices of selected experts for each token.
token_final_scales (torch.Tensor) – Scaling factors for each token’s expert outputs.
fc1_expert_weights (torch.Tensor) – GEMM1 weights for each expert.
fc2_expert_weights (torch.Tensor) – GEMM2 weights for each expert.
output_dtype (torch.dtype) – Desired output data type.
quant_scales (List[torch.Tensor]) –
Quantization scales for the operation.
- NVFP4:
gemm1 activation global scale
gemm1 weights block scales
gemm1 dequant scale
gemm2 activation global scale
gemm2 weights block scales
gemm2 dequant scale
- FP8:
gemm1 dequant scale
gemm2 activation quant scale
gemm2 dequant scale
gemm1 input dequant scale
- Humming FP8 x MXFP4 (
use_wfp4afp8_humming=True): gemm1 folded weight block scales
gemm1 per-local-expert residual scale, including the fixed
2^6compensationreserved scalar or per-local-expert gemm2 activation scale
gemm2 folded weight block scales
gemm2 per-local-expert residual scale, including the fixed
2^6compensation
fc1_expert_biases (Optional[torch.Tensor]) – GEMM1 biases for each expert.
fc2_expert_biases (Optional[torch.Tensor]) – GEMM2 biases for each expert.
input_sf (Optional[torch.Tensor]) – Input scaling factor for quantization.
swiglu_alpha (Optional[torch.Tensor]) – Per-expert float32 SwiGLU alpha. Passing any of
swiglu_alpha,swiglu_betaorswiglu_limitwithActivationType.Swigluselects theSwigluBiaskernel, which computesgate * sigmoid(alpha * gate) * (linear + beta)withlinearclamped to[-limit, limit]andgatecapped atlimit(defaultsalpha=1,beta=0,limit=inf).swiglu_beta (Optional[torch.Tensor]) – Per-expert float32 SwiGLU beta (offset added to the linear half).
swiglu_limit (Optional[torch.Tensor]) – Per-expert float32 SwiGLU clamp limit. A trainer-side clamp value
vand linear offseto(Megatron-Core’sactivation_func_clamp_value/glu_linear_offset) map toswiglu_limit=v,swiglu_beta=oandswiglu_alpha=None.clamped_relu2_limit (Optional[torch.Tensor]) – Model-wide positive clamp scale for ClampedRelu2, provided as a one-element float32 CUDA tensor.
situ_beta (Optional[torch.Tensor]) – Per-expert
betatanh scale for theSituactivation (float32,[num_experts_on_rank]).NoneusesDEFAULT_SITU_BETA.situ_linear_beta (Optional[torch.Tensor]) – Per-expert
linear_betatanh scale for theSituactivation (float32,[num_experts_on_rank]).NoneusesDEFAULT_SITU_LINEAR_BETA.tp_size (int = 1) – Tensor parallelism size. Defaults to 1.
tp_rank (int = 0) – Tensor parallelism rank. Defaults to 0.
ep_size (int = 1) – Expert parallelism size. Defaults to 1.
ep_rank (int = 0) – Expert parallelism rank. Defaults to 0.
cluster_size (int = 1) – Cluster size. Defaults to 1.
cluster_rank (int = 0) – Cluster rank. Defaults to 0.
output (Optional[torch.Tensor] = None) – The output tensor, if not provided, will be allocated internally.
enable_alltoall (bool = False) – Whether to enable all-to-all communication for expert outputs. Defaults to False.
use_deepseek_fp8_block_scale (bool = False) – Whether to use FP8 block scaling. Defaults to False.
use_w4_group_scaling (bool = False) – Whether to use W4A8 group scaling. Defaults to False.
use_mxfp8_act_scaling (bool = False) – Whether to use MXFP8 activation scaling. Defaults to False.
min_latency_mode (bool = False) – Whether to use minimum latency mode. Defaults to False.
use_packed_weights (bool = False) – Whether to use packed uint4x2 weights passed as packed uint8 values. Defaults to False.
use_wfp4afp8_humming (bool = False) – Selects the Humming-style MXFP4-weight x FP8-activation Hopper path with pre-MMA E8M0 scale fusion. This flag is separate from W4A16 because both paths use uint8 FP4 weight storage and
use_w4_group_scaling=True.tune_max_num_tokens (int = 8192) – Maximum number of tokens for tuning. Defaults to 8192.
enable_pdl (Optional[bool]) – Whether to launch the kernel with Programmatic Dependent Launch (PDL).
None(default) lets the runtime pick a safe value based on the device and surrounding stream operations; passTrueto force PDL when every adjacent kernel on the stream also supports it, orFalseto disable.activation_type (ActivationType = ActivationType.Swiglu) – Activation to apply after GEMM1.
Relu2andClampedRelu2use a non-gated GEMM1. On SM10x,ClampedRelu2fuses(limit * tanh(relu(x) / limit))**2into the BF16xBF16 or MXFP8xMXFP8 GEMM1 epilogue.swizzled_input_sf (bool = True) – Whether the input scaling factor (input_sf) is in swizzled layout. Defaults to True. Set to False when input_sf is in linear layout, e.g. after FP4 allgather/alltoall communication where the scaling factors are received in linear (non-swizzled) format. Only relevant when input_sf is not None.
use_fused_finalize (Optional[bool]) – Whether to fuse the top-k expert reduction (“finalize”) into the GEMM2 epilogue. The fused epilogue reduces expert outputs via non-associative atomics, so results are not deterministic run-to-run. Set to False to use the non-fused, deterministic finalize path. When
None(default) the fused path is used unlesstorch.are_deterministic_algorithms_enabled()is true, in which case the autotuner never offers or replays a fused-finalize GEMM2 tactic. An explicit True or False always wins. The flag is part of the autotune cache key, so pass the same resolved value tocutlass_fused_moe_workspace_size().profile_ids (Optional[List[int]]) – Optional
[gemm1_profile, gemm2_profile]override. Both values are absolute indices in the runner’s combined tactic list;-1keeps the default tactic for that GEMM.workspace_buffer (Optional[torch.Tensor]) –
Pre-allocated scratch buffer reused across calls to eliminate per-call workspace allocation (which can reach 10-20 GiB at large batch size). If
None(default), the workspace is allocated and freed on every call. To opt in:Query the required size once at model-load time:
ws_bytes = cutlass_fused_moe_workspace_size( max_num_tokens, hidden_size, intermediate_size, num_experts_total, top_k, x_dtype=..., weight_dtype=..., use_fused_finalize=<same as here>, device=input.device) workspace = torch.empty(ws_bytes, dtype=torch.uint8, device=input.device)
Pass
workspace_buffer=workspaceon every forward call.
The buffer must be a 1-D
torch.uint8ortorch.int8tensor on the same CUDA device as the input, with at leastws_bytesbytes and 128-byte-aligned data pointer (torch.emptyon CUDA satisfies all of these). Allocate one buffer per CUDA stream context; overlapping micro-batches on separate streams each need their own buffer. A buffer sized for the maximum token count is valid for all smaller counts on the same call.backend (str) –
"cutlass"preserves the existing backend."cake"selects the prepared TP8-local SiTU complete call; output and workspace are required. See the Cake SiTU guide for its TRTLLM shuffled NVFP4 weight layout.nvfp4_4over6 (NVFP44Over6Config or None) –
NVFP4 4over6 recipe for the activation quantizations inside the MoE (the GEMM1 input when
inputis unquantized BF16, and the GEMM2 input). NVFP4×NVFP4 only;backend="cake"does not take it.omitted (the default): read the legacy
FLASHINFER_NVFP4_4OVER6*environment variables.None: 4over6 off, environment ignored.NVFP44Over6Config: that recipe. The caller’sfc2_act_global_scale, andfc1_act_global_scalewheninputis BF16, must be built for the samee4m3_max.
- Returns:
out – Output tensor of shape [seq_len, hidden_size].
- Return type:
torch.Tensor
- Raises:
NotImplementedError: – If any of the following features are requested but not implemented: - Minimum Latency Mode
Note
The function supports various data types including FP32, FP16, BF16, FP8, and NVFP4.
It implements both tensor parallelism and expert parallelism.
- Currently, some advanced features like FP8 block scaling and minimum latency mode
are not implemented for Blackwell architecture.