flashinfer.moe_ep.MoEEpAutoLayer

class flashinfer.moe_ep.MoEEpAutoLayer(bootstrap: BootstrapConfig, fleet_params: FleetParams, weights: MoEWeightPack, config: AutoConfig, fleet_knobs: Sequence[AlgoKnob] = ())

Bases: Module

Tune explicit Mega and split candidates over the complete EP forward.

Construct with canonical BF16 weights and candidates implementing the same quantization, activation and expert geometry. Inputs and routing tensors must be contiguous; expert IDs use int32/int64 and routing weights float32. FC1 rows use [up, gate] order; candidate preparation converts the backend layout. Each split candidate pins its transport. Eligible compute tactics use dispatch, compute and combine, without a preliminary compute-only search.

Call warmup collectively in autotune_v2() before capture. A disabled context loads a compatible plan or prepares the first eligible candidate’s fallback. A workspace accepts live row counts up to its agreed runtime_max_tokens_per_rank, or capacity when that hint is omitted. Ordinary serving reuses its plan without selector collectives. An enabled tuning context selects again for a changed live workload. Explicit workspaces let the caller release old captures and their storage independently.

create_workspace(tensors: MoEEpTensors, *, max_tokens_per_rank: int | None = None) → MoEEpAutoWorkspace

Collectively prepare a row bound using the current v2 tuning controls.

warmup(tensors: MoEEpTensors, *, workspace: MoEEpAutoWorkspace | None = None) → None

Prepare and execute a profile collectively, outside graph capture.

forward(tensors: MoEEpTensors, *, workspace: MoEEpAutoWorkspace | None = None) → Tensor

Execute the prepared plan; the workspace owns the returned output.

Pass tensors.output for caller-owned output. Otherwise the next call on this workspace may overwrite the returned tensor.

get_selected_plan(*, workspace: MoEEpAutoWorkspace | None = None) → dict

Return the candidate index and its fixed variant/knobs or compute tactic.

destroy()

Release this layer after retiring every graph that references it.