flashinfer.fused_moe.cute_dsl_fused_moe¶
- flashinfer.fused_moe.cute_dsl_fused_moe(x: ~torch.Tensor, x_sf: ~torch.Tensor | None, token_selected_experts: ~torch.Tensor, token_final_scales: ~torch.Tensor, w1_weight: ~torch.Tensor, w1_weight_sf: ~torch.Tensor, w1_alpha: ~torch.Tensor, fc2_input_scale: ~torch.Tensor | None, w2_weight: ~torch.Tensor, w2_weight_sf: ~torch.Tensor, w2_alpha: ~torch.Tensor, num_experts: int, top_k: int, num_local_experts: int | None = None, local_expert_offset: int = 0, output_dtype: ~torch.dtype = torch.bfloat16, use_fused_finalize: bool = True, moe_output: ~torch.Tensor | None = None, aux_stream: ~torch.cuda.streams.Stream | None = None, enable_pdl: bool = True, activation_type: int = 3, swiglu_alpha: float = 1.0, swiglu_beta: float = 0.0, swiglu_limit: float = 3.4028234663852886e+38, situ_beta: float | None = None, situ_linear_beta: float | None = None, *, quant_mode: str = 'w4a4', per_token_scale: ~torch.Tensor | None = None, tactic: ~typing.Tuple | None = None, localized_weights: list | None = None, localized_streams: list | None = None, localized_sm_count: int | None = None, localized_memset_stream: ~torch.cuda.streams.Stream | None = None, localized_allow_nonlocalized: bool = False, autotune_use_cuda_graph: bool = False, nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>, clamped_relu2_limit: ~torch.Tensor | None = None, swap_ab: bool = False, routing_weights_before_fc2: bool = False) Tensor¶
Run a fused MoE forward pass using CuTe-DSL block-scaled kernels.
Supported architectures: SM100, SM103, and SM107. W4A8 and W8A8 are limited to SM100 and SM103. This is the simple functional API; for CUDA-graph support use
CuteDslMoEWrapperinstead.Auto-tuning is controlled by the
autotune()context manager:with autotune(True): output = cute_dsl_fused_moe(...)
- Parameters:
x (torch.Tensor) – Packed NVFP4 input for
quant_mode="w4a4", MXFP8 input forquant_mode="w4a8"or"mxfp8", or BF16 input forquant_mode="w4a16".x_sf (Optional[torch.Tensor]) – Scale factors for W4A4, W4A8, or W8A8; must be
Noneforquant_mode="w4a16".token_selected_experts (torch.Tensor) – Expert assignments of shape
[num_tokens, top_k].token_final_scales (torch.Tensor) – Routing weights of shape
[num_tokens, top_k].w1_weight (torch.Tensor) – GEMM1 weights (gate + up fused for gated activations, or a single projection for non-gated activations).
w1_weight_sf (torch.Tensor) – Scale factors for
w1_weight.w1_alpha (torch.Tensor) – Per-expert global scale for GEMM1.
fc2_input_scale (Optional[torch.Tensor]) – Global scale for W4A4 GEMM2 input quantization; must be
Nonefor W4A8, W8A8, and W4A16.w2_weight (torch.Tensor) – GEMM2 weights (down projection).
w2_weight_sf (torch.Tensor) – Scale factors for
w2_weight.w2_alpha (torch.Tensor) – Per-expert global scale for GEMM2.
num_experts (int) – Total number of experts.
top_k (int) – Number of experts routed to per token.
num_local_experts (Optional[int]) – Local experts for expert parallelism. Defaults to
num_experts.local_expert_offset (int) – Offset of local experts in the global expert space. Defaults to
0.output_dtype (torch.dtype) – Output dtype. Defaults to
torch.bfloat16.use_fused_finalize (bool) – Use atomic fused finalize; otherwise use the deterministic two-stage finalize. Defaults to
True.moe_output (Optional[torch.Tensor]) – Pre-allocated output buffer. Allocated internally if
None.aux_stream (Optional[torch.cuda.Stream]) – Optional auxiliary CUDA stream used to overlap setup work with the main computation.
enable_pdl (bool) – Enable Programmatic Dependent Launch. Defaults to
True.activation_type (int) – FC1 activation type. Use
ActivationType.Swiglufor gated SwiGLU/SiTU,ActivationType.GegluTanhfor tanh-approximate GeGLU,ActivationType.Relu2for non-gated ReLU^2, andActivationType.ClampedRelu2for non-gated clamped ReLU^2 (requiresclamped_relu2_limit). Settingsitu_betaselects SiTU;swiglu_oaiis represented asActivationType.Swigluwith non-defaultswiglu_alpha/beta/limit.swiglu_alpha (float) – SwiGLU parameters.
swiglu_beta (float) – SwiGLU parameters.
swiglu_limit (float) – SwiGLU parameters.
quant_mode (str) – Compute mode:
"w4a4","w4a8","w4a16", or"mxfp8". Defaults to"w4a4"."nvfp4"is a deprecated alias for"w4a4"."mxfp8"runs MXFP8 weights (float8_e4m3fnwith E8M0 block-32 scales in the MMA layout) against MXFP8 activations (xisfloat8_e4m3fnandx_sfthe row-major E8M0 scales asuint8), withfc2_input_scale=Noneand BF16 output. W4A8 and MXFP8 support SiTU and either finalize mode on Blackwell (SM100/SM103) withswap_ab=True. The existing path requires fused finalize and does not support SiTU. Neither mode supports Rubin or per-token scales.situ_beta (Optional[float]) – When set with
ActivationType.Swiglu, use the SiTU gatebeta * tanh(gate / beta) * sigmoid(gate).situ_linear_beta (Optional[float]) – Optional SiTU tanh clamp for the up branch.
clamped_relu2_limit (Optional[torch.Tensor]) – Model-wide positive clamp scale
LforActivationType.ClampedRelu2, provided as a one-element float32 CUDA tensor of shape(1,)on the device ofx, as incutlass_fused_moe(). ClampedRelu2 computesrelu(L * tanh(x / L)) ** 2on the alpha-scaled FC1 outputx(equivalently(L * tanh(relu(x) / L)) ** 2, the CUTLASS MoEClampedRelu2). The kernel reads the tensor at runtime, so the compiled kernel does not depend on its value. Required withClampedRelu2and rejected for every other activation. Supported by"w4a4","w4a8"and"mxfp8"; not by"w4a16"or on SM107.per_token_scale (Optional[torch.Tensor]) – Optional W4A4 per-token input row scale for GEMM1.
nvfp4_4over6 (NVFP44Over6Config or None) –
NVFP4 4over6 recipe for the W4A4 per-token requantization of the GEMM2 input (the GEMM1 input is quantized by the caller).
omitted (the default): read the legacy
FLASHINFER_NVFP4_4OVER6*environment variables.None: 4over6 off, environment ignored.NVFP44Over6Config: that recipe. Requiresquant_mode="w4a4"withper_token_scale, otherwiseValueError;fc2_input_scalemust have been built for the same recipe.
tactic (Optional[Tuple]) – Tactic tuple, or
Nonefor auto-selection via the runtime tuner.localized_weights (Optional[list]) –
Per-locality-domain W4A4 weight-shard dictionaries. Supplying these enables localized execution; exactly two equal-width shards are required. Entry
icontainsw1_weight,w1_weight_sf,w2_weight, andw2_weight_sfforlocalized_streams[i]. Allocate all four tensors in the corresponding locality-domain memory pool, for example insidetorch.cuda.use_mem_pool(domain_pool); slicing an allocation from another pool does not relocate its storage.For gated W4A4 with E local experts, hidden size H, and intermediate size I, packed uint8 full weights have shapes
[E, 2*I, H/2](FC1) and[E, H, I/2](FC2). Split dimension 1 into two contiguous halves: each domain receives[E, I, H/2]and[E, H/2, I/2]. Keep each interleaved 64-row up/gate pair together in FC1. Split the corresponding unswizzled scale rows identically, then swizzle each shard independently into the CuTe MMA scale layout. FC2 retains the full reduction dimension; its shards produce disjoint hidden-output columns.Continue passing the original full-width
w1_weight,w2_weight, and their*_sftensors to the regular arguments; do not passNoneor shard tensors there. Withlocalized_allow_nonlocalized=False, neither the GEMMs nor the autotuner read them, so a placeholder that keeps the full-width metadata but releases its storage (for examplew.untyped_storage().resize_(0)) is valid; onlyFLASHINFER_LOGLEVEL=5statistics would touch its data. WithTrue, retain their valid contents because autotuning may run and select the full-width path.w1_alpha,w2_alpha, andfc2_input_scaleremain shared across domains.Example with shards already allocated in their domain pools and scale factors already converted to the MMA layout:
shards = [ { "w1_weight": w1_domain0, "w1_weight_sf": w1_sf_domain0, "w2_weight": w2_domain0, "w2_weight_sf": w2_sf_domain0, }, { "w1_weight": w1_domain1, "w1_weight_sf": w1_sf_domain1, "w2_weight": w2_domain1, "w2_weight_sf": w2_sf_domain1, }, ]
localized_streams (Optional[list]) – One long-lived green-context CUDA stream for each locality domain (for example
GreenContext.Stream()). Work is forked from and joined to the caller’s current stream withtorch.cuda.execute_on_streams. Localized execution requires a PyTorch build providing this API and raisesRuntimeErrorif it is unavailable. Non-localized execution does not require this API.localized_sm_count (Optional[int]) – Number of SMs available to each locality-domain stream, used to size the persistent kernel grid.
localized_memset_stream (Optional[torch.cuda.Stream]) – Optional CUDA stream used to overlap fused-finalize output zeroing with FC1.
localized_allow_nonlocalized (bool) – Include the full-width path in autotuning when localized weights are supplied. The caller must retain usable full-width weights.
swap_ab (bool) – Select the optimized SM100/SM103 implementation. Gated FC1 weights and scales must use up16/gate16 rows when True, or the existing up64/gate64 rows when False (default). W4A16 always uses weights on A and activations on B; False handles the original 64-row layout with an in-kernel transpose.
autotune_use_cuda_graph (bool) – Profile tactics through CUDA Graph replay.
routing_weights_before_fc2 (bool) – Apply routing weights after ClampedRelu2 and before NVFP4 quantization of the GEMM2 input. Rounds to BF16 after GEMM1, soft clamping, squaring, and routing-weight multiplication. Requires W4A4, BF16 output and
swap_ab=Falseon SM100/SM103. Per-token scales and localized weights are unsupported. This path uses a single-CTA 128x128 tactic on one stream without PDL.
- Returns:
Output tensor of shape
[num_tokens, hidden_size].- Return type:
torch.Tensor