flashinfer.fused_moe.prims_ts_fp8_block_scale_routed_moe

flashinfer.fused_moe.prims_ts_fp8_block_scale_routed_moe(topk_ids: Tensor | Tuple[Tensor, Tensor], routing_bias: Tensor | None, hidden_states: Tensor, hidden_states_scale: Tensor, gemm1_weights: Tensor, gemm1_weights_scale: Tensor, gemm2_weights: Tensor, gemm2_weights_scale: Tensor, num_experts: int, top_k: int, n_group: int | None, topk_group: int | None, intermediate_size: int, local_expert_offset: int, local_num_experts: int, routed_scaling_factor: float | None, routing_method_type: int = 0, use_shuffled_weight: bool = False, weight_layout: int = 0, do_finalize: bool = True, enable_pdl: bool | None = None, gemm1_lora_delta: Tensor | None = None, output: Tensor | None = None, tune_max_num_tokens: int = 8192, fp8_quantization_type: Fp8QuantizationType = Fp8QuantizationType.DeepSeekFp8, activation_type: int = 3, gemm1_alpha: Tensor | None = None, gemm1_beta: Tensor | None = None, gemm1_clamp_limit: Tensor | None = None, *, gemm1_bias: Tensor | None = None, gemm2_bias: Tensor | None = None) → List[Tensor] | Tensor

Pre-routed FP8 block-scaled MoE using the Prims-TS backend on SM100.

Same arguments and return value as trtllm_fp8_block_scale_routed_moe(), plus optional keyword-only FC1/FC2 bias tensors.

Parameters:

  • topk_ids (torch.Tensor or Tuple[torch.Tensor, torch.Tensor]) – Packed (expert_id, weight) tensor or unpacked (topk_ids, topk_weights) pair.

  • routing_bias (Optional[torch.Tensor]) – Optional [num_experts] routing bias.

  • hidden_states (torch.Tensor) – Activations (BF16/FP16 or float8_e4m3fn depending on mode).

  • hidden_states_scale (torch.Tensor) – Block scales for hidden_states.

  • gemm1_weights (torch.Tensor) – FC1 expert weights.

  • gemm1_weights_scale (torch.Tensor) – FC1 block scales.

  • gemm2_weights (torch.Tensor) – FC2 expert weights.

  • gemm2_weights_scale (torch.Tensor) – FC2 block scales.

  • num_experts (int) – Total number of experts.

  • top_k (int) – Experts selected per token.

  • n_group (Optional[int]) – Number of expert groups.

  • topk_group (Optional[int]) – Groups considered for top-k routing.

  • intermediate_size (int) – Intermediate (FFN) width.

  • local_expert_offset (int) – Global offset of the first local expert.

  • local_num_experts (int) – Number of experts resident on this device.

  • routed_scaling_factor (Optional[float]) – Optional routing scale.

  • routing_method_type (int) – Routing method selector (default 0).

  • use_shuffled_weight (bool) – Whether weights use the shuffled layout (default False).

  • weight_layout (int) – Weight layout enum value (default MajorK).

  • do_finalize (bool) – If True, return the finalized MoE output.

  • enable_pdl (Optional[bool]) – Enable Programmatic Dependent Launch when supported.

  • gemm1_lora_delta (Optional[torch.Tensor]) – Optional MoE LoRA delta applied before the gated activation.

  • output (Optional[torch.Tensor]) – Optional in-place output tensor.

  • tune_max_num_tokens (int) – Autotune token-bucket upper bound (default 8192).

  • fp8_quantization_type (Fp8QuantizationType) – Block-scale recipe (DeepSeek FP8 or MXFP8).

  • activation_type (int) – Activation enum value (default Swiglu).

  • gemm1_alpha (Optional[torch.Tensor]) – Optional per-expert SwiGLU alpha.

  • gemm1_beta (Optional[torch.Tensor]) – Optional per-expert SwiGLU beta.

  • gemm1_clamp_limit (Optional[torch.Tensor]) – Optional per-expert clamp limit.

  • gemm1_bias (Optional[torch.Tensor]) – Optional FC1 bias (keyword-only).

  • gemm2_bias (Optional[torch.Tensor]) – Optional FC2 bias (keyword-only).

Returns:

Same return contract as trtllm_fp8_block_scale_routed_moe().

Return type:

torch.Tensor or List[torch.Tensor]