flashinfer.fused_moe.trtllm_fp4_block_scale_routed_moe¶
- flashinfer.fused_moe.trtllm_fp4_block_scale_routed_moe(topk_ids: ~torch.Tensor | ~typing.Tuple[~torch.Tensor, ~torch.Tensor], routing_bias: ~torch.Tensor | None, hidden_states: ~torch.Tensor, hidden_states_scale: ~torch.Tensor | None, gemm1_weights: ~torch.Tensor, gemm1_weights_scale: ~torch.Tensor, gemm1_bias: ~torch.Tensor | None, gemm1_alpha: ~torch.Tensor | None, gemm1_beta: ~torch.Tensor | None, gemm1_clamp_limit: ~torch.Tensor | None, gemm2_weights: ~torch.Tensor, gemm2_weights_scale: ~torch.Tensor, gemm2_bias: ~torch.Tensor | None, output1_scale_scalar: ~torch.Tensor | None, output1_scale_gate_scalar: ~torch.Tensor | None, output2_scale_scalar: ~torch.Tensor | None, num_experts: int, top_k: int, n_group: int | None, topk_group: int | None, intermediate_size: int, local_expert_offset: int, local_num_experts: int, routed_scaling_factor: float | None, routing_method_type: int = 0, do_finalize: bool = True, enable_pdl: bool | None = None, activation_type: int = 3, per_token_scale: ~torch.Tensor | None = None, output: ~torch.Tensor | None = None, tune_max_num_tokens: int = 8192, gemm1_lora_delta: ~torch.Tensor | None = None, valid_hidden_size: int | None = None, valid_intermediate_size: int | None = None, num_fused_shared_experts: int | None = None, hidden_states_scale_layout: ~flashinfer.tllm_enums.SfLayout | None = None, workspace_buffer: ~torch.Tensor | None = None, nvfp4_4over6: ~flashinfer.quantization.nvfp4_quantization_utils.NVFP44Over6Config | None = <nvfp4_4over6 unset: read FLASHINFER_NVFP4_4OVER6*>, routing_metadata: ~flashinfer.fused_moe.core.TrtllmMoERoutingMetadataSlot | None = None) List[Tensor]¶
FP4 block scale MoE operation with pre-computed routing.
This function supports two pre-computed routing formats: 1. Packed format:
topk_idsis a single int32 tensor with(expert_id << 16) | weightentries (high 16 bits = int16 expert id, low 16 bits = float16/bfloat16 weight, matchingPackedScoreIdxininclude/flashinfer/trtllm/fused_moe/RoutingKernel.h).Unpacked format:
topk_idsis a tuple(topk_ids, topk_weights).
- Parameters:
topk_ids (Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]) – Pre-computed routing decision. Either a single int32 tensor of shape
[seq_len, top_k]in packed format(expert_id << 16) | weightor a tuple(ids, weights)whereidsis int32 of shape[seq_len, top_k](plain expert indices) andweightsisbfloat16orfloat32of the same shape (routing weights). The weights are consumed at their native dtype (no cast), so passing thefloat32weights emitted by typical routers is copy-free.routing_bias (Optional[torch.Tensor]) –
[num_experts]routing bias,bfloat16orfloat32. May beNone.hidden_states (torch.Tensor) – Hidden states of shape
[seq_len, hidden_size // 2](NVFP4) or[seq_len, hidden_size](MXFP8 / bfloat16). Supports bfloat16, MXFP8 (float8_e4m3fn), and NVFP4 (packed intouint8).hidden_states_scale (Optional[torch.Tensor]) –
[seq_len, hidden_size // (32 if mxfp8 else 16)]block scales of the hidden states, float8. The equivalent flat[seq_len * hidden_size // (32 if mxfp8 else 16)]buffer returned bymxfp8_quantize()withis_sf_swizzled_layout=False(the linear layout) is also accepted. Declare which layout the buffer is in withhidden_states_scale_layout; only the linear layout is supported.gemm1_weights (torch.Tensor) –
[num_experts, 2 * intermediate_size, hidden_size // 2]packed FP4 FC1 weights,uint8.gemm1_weights_scale (torch.Tensor) –
[num_experts, 2 * intermediate_size, hidden_size // (32 if mxfp4 else 16)]FC1 weight block scales, float8.gemm1_bias (Optional[torch.Tensor]) –
[num_experts, 2 * intermediate_size]FC1 bias, float32.gemm1_alpha (Optional[torch.Tensor]) –
[num_experts]swiglu alpha, float32. For SiTU this is[local_num_experts], finite and positive;Nonematerializes per-expertalpha=1.gemm1_beta (Optional[torch.Tensor]) –
[num_experts]swiglu beta, float32. For SiTU this is[local_num_experts], finite and positive;Nonematerializes per-expertbeta=1.gemm1_clamp_limit (Optional[torch.Tensor]) –
[num_experts]swiglu clamp limit, float32. ForSwigluStep, pass the physical limit divided byoutput1_scale_gate_scalarwhen the scale is non-unit;Noneuses physical7. Alpha and beta are not accepted. For SiTU a provided limit is per-local-expert, finite, and positive; it clampsx0to[-limit, limit]andx1from above.gemm2_weights (torch.Tensor) –
[num_experts, hidden_size, intermediate_size]packed FP4 FC2 weights,uint8.gemm2_weights_scale (torch.Tensor) –
[num_experts, hidden_size, intermediate_size // (32 if mxfp4 else 16)]FC2 weight block scales, float8.gemm2_bias (Optional[torch.Tensor]) –
[num_experts, hidden_size]FC2 bias, float32.output1_scale_scalar (Optional[torch.Tensor]) –
[local_num_experts]scaling factors for the first-layer activation output.output1_scale_gate_scalar (Optional[torch.Tensor]) –
[local_num_experts]scaling factors for the first-layer gate output.output2_scale_scalar (Optional[torch.Tensor]) –
[local_num_experts]scaling factors for the second-layer output.num_experts (int) – Total number of experts.
top_k (int) – Number of experts to route to per token.
n_group (Optional[int]) – Number of expert groups.
topk_group (Optional[int]) – Number of groups to consider for top-k routing.
intermediate_size (int) – Size of the intermediate layer.
local_expert_offset (int) – Offset of local experts in the global expert space.
local_num_experts (int) – Number of experts handled by this device.
routed_scaling_factor (Optional[float]) – Scaling factor for routing.
routing_method_type (int) –
Routing method (default
0). Selects the routing-kernel pipeline; matchesflashinfer.tllm_enums.RoutingMethodType.0Default— Softmax → TopK.1Renormalize— TopK → Softmax.2DeepSeekV3— Sigmoid → RoutingBiasAdd → Top-2 in group → Top-topk_groupgroups → Top-top_kexperts from the selected groups.3Llama4— Top-1 → Sigmoid.4RenormalizeNaive— Softmax → TopK → Renormalize (Qwen3 style).5TopK— TopK only (no softmax/sigmoid).6SigmoidRenorm— Sigmoid → TopK → Renormalize (divide by the sum of the top-K weights).7MiniMax2— Sigmoid + Bias → TopK → ScaledSumNormalize (routeScale = 1.0,epsilon = 1e-20).8Sigmoid— Sigmoid → TopK (no renormalization).9TopKSigmoid— TopK → Sigmoid (no renormalization).10Unspecified— reserved.
do_finalize (bool) – Whether to finalize the output (default
True).enable_pdl (Optional[bool]) – Whether to enable Programmatic Dependent Launch.
activation_type (int) – Activation type (default
3— Swiglu).10SiTU computesbeta*tanh(x0/beta) * alpha*tanh(x1/alpha)*sigmoid(x1).per_token_scale (Optional[torch.Tensor]) –
[seq_len]per-token scaling factors, float32.output (Optional[torch.Tensor]) – Optional in-place
[seq_len, hidden_size]output tensor.tune_max_num_tokens (int) – Maximum number of tokens for autotuning (default
8192).gemm1_lora_delta (Optional[torch.Tensor]) – Optional MoE LoRA delta of shape
[num_tokens, top_k, 2 * intermediate_size],bfloat16. When set it is added to FC1 before the fused gated activation and the post-activation FC1 output is appended to the return list.hidden_states_scale_layout (Optional[flashinfer.tllm_enums.SfLayout]) –
Layout of
hidden_states_scale. Seetrtllm_fp4_block_scale_moe()— validation only, linear layout only, and omitting it is deprecated.Deprecated since version Omitting: this argument is deprecated.
None(the default) inferslayout_linearand emits aDeprecationWarning. Both the inference and this argument’s optionality are deprecated together.valid_hidden_size (Optional[int]) – Valid (unpadded) hidden dimension. When provided, the
hidden_sizeimplied by the tensor shapes is treated as padded and only the valid region is contracted. Matching TensorRT-LLM, the returned tensor is exactlyvalid_hidden_sizewide: GEMM2 computesroundUp(valid_hidden_size, 128)columns (that is what the FC2 weights, weight scales and bias are sized for) and finalize writes only the leadingvalid_hidden_sizeof them. A caller-suppliedoutputmust therefore havevalid_hidden_sizecolumns. For NVFP4 the paddedhidden_sizemust be a multiple of 256 (128 gives wrong results). DefaultNone(use the fullhidden_size).valid_intermediate_size (Optional[int]) – Valid (unpadded) intermediate dimension. When provided,
intermediate_sizeis treated as padded and only the valid region is computed. DefaultNone(use the fullintermediate_size).num_fused_shared_experts (Optional[int]) – Number of shared experts to fuse into the MoE kernel (default
None/0). When> 0, every per-expert tensor must havenum_experts + num_fused_shared_expertsrows in the expert dimension — the shared-expert weights are appended after the routed ones. Every token is unconditionally routed to the shared experts with weight1.0. Expert parallelism is not yet supported together with fused shared experts: requirelocal_expert_offset == 0andlocal_num_experts == num_experts. OnlyDeepSeekV3routing is supported when this is> 0.workspace_buffer (Optional[torch.Tensor]) – Caller-owned 1-D
uint8CUDA buffer of at leasttrtllm_fused_moe_workspace_size()bytes. When given, the launch carves all of its internal scratch (routing maps, intermediate GEMM outputs and batched-GEMM workspaces) from it instead of allocating per call, so MoE layers that run sequentially on one stream can share a single buffer. Requiresdo_finalize=Trueand nogemm1_lora_delta. Autotuning profiles from the same buffer; scratch that does not fit (a larger token count) is allocated per call. DefaultNone(allocate per call).nvfp4_4over6 (NVFP44Over6Config or None) –
NVFP4 4over6 recipe for the per-token requantization of the GEMM2 input (the GEMM1 input is quantized by the caller).
omitted (the default): read the legacy
FLASHINFER_NVFP4_4OVER6*environment variables.None: 4over6 off, environment ignored.NVFP44Over6Config: that recipe. Requiresper_token_scale; no other path has a 4over6 requantization, so a pinned recipe raisesValueError.
routing_metadata (Optional[TrtllmMoERoutingMetadataSlot]) – A slot of routing metadata allocated with
trtllm_moe_allocate_routing_metadata_multi_tile()and already built fromtopk_idsbypopulate_trtllm_moe_routing_metadata_(). When given, the call skips the routing kernel and launches only the expert GEMMs with the slot’s tile, so the routing can run earlier, for example on another stream while the activations are quantized. The caller must populate the slot from the sametopk_idsand order that population before this call. Autotuning still profiles the ordinary routed operation, routing included; the autotuned config is used only when the tuner chose the slot’s tile, otherwise that tile’s default config runs. The body allocates its own buffers. Not supported withgemm1_lora_delta, fused shared experts orworkspace_buffer(ValueError). DefaultNone(route inside the call).
- Returns:
Return shape depends on
do_finalizeandgemm1_lora_delta; seetrtllm_bf16_routed_moe()for the table.- Return type:
torch.Tensor or List[torch.Tensor]