flashinfer.fused_moe.cute_dsl_fused_moe

flashinfer.fused_moe.cute_dsl_fused_moe(x: ~torch.Tensor, x_sf: ~torch.Tensor | None, token_selected_experts: ~torch.Tensor, token_final_scales: ~torch.Tensor, w1_weight: ~torch.Tensor, w1_weight_sf: ~torch.Tensor, w1_alpha: ~torch.Tensor, fc2_input_scale: ~torch.Tensor | None, w2_weight: ~torch.Tensor, w2_weight_sf: ~torch.Tensor, w2_alpha: ~torch.Tensor, num_experts: int, top_k: int, num_local_experts: int | None = None, local_expert_offset: int = 0, output_dtype: ~torch.dtype = torch.bfloat16, use_fused_finalize: bool = True, moe_output: ~torch.Tensor | None = None, aux_stream: ~torch.cuda.streams.Stream | None = None, enable_pdl: bool = True, activation_type: int = 3, swiglu_alpha: float = 1.0, swiglu_beta: float = 0.0, swiglu_limit: float = 3.4028234663852886e+38, situ_beta: float | None = None, situ_linear_beta: float | None = None, *, quant_mode: str = 'w4a4', per_token_scale: ~torch.Tensor | None = None, tactic: ~typing.Tuple | None = None, localized_weights: list | None = None, localized_streams: list | None = None, localized_sm_count: int | None = None, localized_memset_stream: ~torch.cuda.streams.Stream | None = None, localized_allow_nonlocalized: bool = False, autotune_use_cuda_graph: bool = False, nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>, clamped_relu2_limit: ~torch.Tensor | None = None, swap_ab: bool = False, routing_weights_before_fc2: bool = False) → Tensor

Run a fused MoE forward pass using CuTe-DSL block-scaled kernels.

Supported architectures: SM100, SM103, and SM107. W4A8 and W8A8 are limited to SM100 and SM103. This is the simple functional API; for CUDA-graph support use CuteDslMoEWrapper instead.

Auto-tuning is controlled by the autotune() context manager:

with autotune(True):
    output = cute_dsl_fused_moe(...)
Parameters:
  • x (torch.Tensor) – Packed NVFP4 input for quant_mode="w4a4", MXFP8 input for quant_mode="w4a8" or "mxfp8", or BF16 input for quant_mode="w4a16".

  • x_sf (Optional[torch.Tensor]) – Scale factors for W4A4, W4A8, or W8A8; must be None for quant_mode="w4a16".

  • token_selected_experts (torch.Tensor) – Expert assignments of shape [num_tokens, top_k].

  • token_final_scales (torch.Tensor) – Routing weights of shape [num_tokens, top_k].

  • w1_weight (torch.Tensor) – GEMM1 weights (gate + up fused for gated activations, or a single projection for non-gated activations).

  • w1_weight_sf (torch.Tensor) – Scale factors for w1_weight.

  • w1_alpha (torch.Tensor) – Per-expert global scale for GEMM1.

  • fc2_input_scale (Optional[torch.Tensor]) – Global scale for W4A4 GEMM2 input quantization; must be None for W4A8, W8A8, and W4A16.

  • w2_weight (torch.Tensor) – GEMM2 weights (down projection).

  • w2_weight_sf (torch.Tensor) – Scale factors for w2_weight.

  • w2_alpha (torch.Tensor) – Per-expert global scale for GEMM2.

  • num_experts (int) – Total number of experts.

  • top_k (int) – Number of experts routed to per token.

  • num_local_experts (Optional[int]) – Local experts for expert parallelism. Defaults to num_experts.

  • local_expert_offset (int) – Offset of local experts in the global expert space. Defaults to 0.

  • output_dtype (torch.dtype) – Output dtype. Defaults to torch.bfloat16.

  • use_fused_finalize (bool) – Use atomic fused finalize; otherwise use the deterministic two-stage finalize. Defaults to True.

  • moe_output (Optional[torch.Tensor]) – Pre-allocated output buffer. Allocated internally if None.

  • aux_stream (Optional[torch.cuda.Stream]) – Optional auxiliary CUDA stream used to overlap setup work with the main computation.

  • enable_pdl (bool) – Enable Programmatic Dependent Launch. Defaults to True.

  • activation_type (int) – FC1 activation type. Use ActivationType.Swiglu for gated SwiGLU/SiTU, ActivationType.GegluTanh for tanh-approximate GeGLU, ActivationType.Relu2 for non-gated ReLU^2, and ActivationType.ClampedRelu2 for non-gated clamped ReLU^2 (requires clamped_relu2_limit). Setting situ_beta selects SiTU; swiglu_oai is represented as ActivationType.Swiglu with non-default swiglu_alpha/beta/limit.

  • swiglu_alpha (float) – SwiGLU parameters.

  • swiglu_beta (float) – SwiGLU parameters.

  • swiglu_limit (float) – SwiGLU parameters.

  • quant_mode (str) – Compute mode: "w4a4", "w4a8", "w4a16", or "mxfp8". Defaults to "w4a4". "nvfp4" is a deprecated alias for "w4a4". "mxfp8" runs MXFP8 weights (float8_e4m3fn with E8M0 block-32 scales in the MMA layout) against MXFP8 activations (x is float8_e4m3fn and x_sf the row-major E8M0 scales as uint8), with fc2_input_scale=None and BF16 output. W4A8 and MXFP8 support SiTU and either finalize mode on Blackwell (SM100/SM103) with swap_ab=True. The existing path requires fused finalize and does not support SiTU. Neither mode supports Rubin or per-token scales.

  • situ_beta (Optional[float]) – When set with ActivationType.Swiglu, use the SiTU gate beta * tanh(gate / beta) * sigmoid(gate).

  • situ_linear_beta (Optional[float]) – Optional SiTU tanh clamp for the up branch.

  • clamped_relu2_limit (Optional[torch.Tensor]) – Model-wide positive clamp scale L for ActivationType.ClampedRelu2, provided as a one-element float32 CUDA tensor of shape (1,) on the device of x, as in cutlass_fused_moe(). ClampedRelu2 computes relu(L * tanh(x / L)) ** 2 on the alpha-scaled FC1 output x (equivalently (L * tanh(relu(x) / L)) ** 2, the CUTLASS MoE ClampedRelu2). The kernel reads the tensor at runtime, so the compiled kernel does not depend on its value. Required with ClampedRelu2 and rejected for every other activation. Supported by "w4a4", "w4a8" and "mxfp8"; not by "w4a16" or on SM107.

  • per_token_scale (Optional[torch.Tensor]) – Optional W4A4 per-token input row scale for GEMM1.

  • nvfp4_4over6 (NVFP44Over6Config or None) –

    NVFP4 4over6 recipe for the W4A4 per-token requantization of the GEMM2 input (the GEMM1 input is quantized by the caller).

    • omitted (the default): read the legacy FLASHINFER_NVFP4_4OVER6* environment variables.

    • None: 4over6 off, environment ignored.

    • NVFP44Over6Config: that recipe. Requires quant_mode="w4a4" with per_token_scale, otherwise ValueError; fc2_input_scale must have been built for the same recipe.

  • tactic (Optional[Tuple]) – Tactic tuple, or None for auto-selection via the runtime tuner.

  • localized_weights (Optional[list]) –

    Per-locality-domain W4A4 weight-shard dictionaries. Supplying these enables localized execution; exactly two equal-width shards are required. Entry i contains w1_weight, w1_weight_sf, w2_weight, and w2_weight_sf for localized_streams[i]. Allocate all four tensors in the corresponding locality-domain memory pool, for example inside torch.cuda.use_mem_pool(domain_pool); slicing an allocation from another pool does not relocate its storage.

    For gated W4A4 with E local experts, hidden size H, and intermediate size I, packed uint8 full weights have shapes [E, 2*I, H/2] (FC1) and [E, H, I/2] (FC2). Split dimension 1 into two contiguous halves: each domain receives [E, I, H/2] and [E, H/2, I/2]. Keep each interleaved 64-row up/gate pair together in FC1. Split the corresponding unswizzled scale rows identically, then swizzle each shard independently into the CuTe MMA scale layout. FC2 retains the full reduction dimension; its shards produce disjoint hidden-output columns.

    Continue passing the original full-width w1_weight, w2_weight, and their *_sf tensors to the regular arguments; do not pass None or shard tensors there. With localized_allow_nonlocalized=False, neither the GEMMs nor the autotuner read them, so a placeholder that keeps the full-width metadata but releases its storage (for example w.untyped_storage().resize_(0)) is valid; only FLASHINFER_LOGLEVEL=5 statistics would touch its data. With True, retain their valid contents because autotuning may run and select the full-width path. w1_alpha, w2_alpha, and fc2_input_scale remain shared across domains.

    Example with shards already allocated in their domain pools and scale factors already converted to the MMA layout:

    shards = [
        {
            "w1_weight": w1_domain0,
            "w1_weight_sf": w1_sf_domain0,
            "w2_weight": w2_domain0,
            "w2_weight_sf": w2_sf_domain0,
        },
        {
            "w1_weight": w1_domain1,
            "w1_weight_sf": w1_sf_domain1,
            "w2_weight": w2_domain1,
            "w2_weight_sf": w2_sf_domain1,
        },
    ]
    

  • localized_streams (Optional[list]) – One long-lived green-context CUDA stream for each locality domain (for example GreenContext.Stream()). Work is forked from and joined to the caller’s current stream with torch.cuda.execute_on_streams. Localized execution requires a PyTorch build providing this API and raises RuntimeError if it is unavailable. Non-localized execution does not require this API.

  • localized_sm_count (Optional[int]) – Number of SMs available to each locality-domain stream, used to size the persistent kernel grid.

  • localized_memset_stream (Optional[torch.cuda.Stream]) – Optional CUDA stream used to overlap fused-finalize output zeroing with FC1.

  • localized_allow_nonlocalized (bool) – Include the full-width path in autotuning when localized weights are supplied. The caller must retain usable full-width weights.

  • swap_ab (bool) – Select the optimized SM100/SM103 implementation. Gated FC1 weights and scales must use up16/gate16 rows when True, or the existing up64/gate64 rows when False (default). W4A16 always uses weights on A and activations on B; False handles the original 64-row layout with an in-kernel transpose.

  • autotune_use_cuda_graph (bool) – Profile tactics through CUDA Graph replay.

  • routing_weights_before_fc2 (bool) – Apply routing weights after ClampedRelu2 and before NVFP4 quantization of the GEMM2 input. Rounds to BF16 after GEMM1, soft clamping, squaring, and routing-weight multiplication. Requires W4A4, BF16 output and swap_ab=False on SM100/SM103. Per-token scales and localized weights are unsupported. This path uses a single-CTA 128x128 tactic on one stream without PDL.

Returns:

Output tensor of shape [num_tokens, hidden_size].

Return type:

torch.Tensor