flashinfer.moe_ep.MoEEpCommParams

class flashinfer.moe_ep.MoEEpCommParams(num_experts: int, top_k: int, max_tokens_per_rank: int, hidden_size: int, dtype: torch.dtype | None = None, dispatch_format: QuantFormat | None = None, invalid_expert_id: int = -1)

Bases: object

EP sizing shared by every communication backend.

max_tokens_per_rank bounds the number of tokens any rank dispatches in one call and sizes the backend’s receive buffers. hidden_size and dtype describe the unquantized token row, which is also what the expert computation hands to combine. dispatch_format is the QuantFormat of the activations as dispatched, values plus per-token scale factors; None dispatches unquantized dtype rows. Backends that reserve buffers per dispatched byte size them from dispatch_bytes_per_token.

property token_dtype: torch.dtype

dtype, defaulting to BF16.

property dispatch_bytes_per_token: int

Bytes of one token’s dispatched activations, scale factors included.