flashinfer.fused_moe.alphamoe_fused_router¶
- flashinfer.fused_moe.alphamoe_fused_router(logits: Tensor, *, top_k: int, block_m: int = 8, has_shared_expert: bool = False, plan: AlphaMoERoutePlan | None = None) AlphaMoERoutePlan¶
Route FP32 logits and build an AlphaMoE aligned plan (SM100/SM103).
Generated kernels perform top-k selection, selected-logit softmax, per-expert counting, block padding, and grouped route scatter. The token count selects the launch: one CTA for at most 8 tokens, one cooperative grid up to 512 tokens, and a cooperative grid followed by a tail kernel above that. Calls with 512 experts and
has_shared_expert=Falseuse dedicated entries for most token counts andtop_kvalues; up to 128 tokens these run as one thread-block cluster without a grid barrier. Calls with 256, 257 or 384 experts,has_shared_expert=False, 129 to 512 tokens andtop_kfrom 8 to 10 use dedicated medium entries. From 129 to 256 tokens, calls with 256, 257, 384 or 512 experts,has_shared_expert=False,top_kfrom 8 to 10 andblock_m5, 8 or 16 (8 or 16 at 256 experts) run as one thread-block cluster. Withhas_shared_expert=True, the last expert is forced into the final route slot and the remainingtop_k - 1experts are selected from[0, num_experts - 1).- Parameters:
logits (torch.Tensor) – Contiguous FP32 CUDA tensor shaped
(num_tokens, num_experts).num_tokensmust be positive andnum_expertsmust be in[1, 512]. Logits must be finite; the result for NaN or infinite logits is unspecified.top_k (int) – Routes per token, in
[1, min(num_experts, 16)].block_m (int) – Per-expert alignment in
[1, 16]. Defaults to 8.has_shared_expert (bool) – Force the last expert into every token’s last route slot. This mode requires
top_k >= 2.plan (Optional[AlphaMoERoutePlan]) – Reusable output/workspace allocation. When omitted, a worst-case-capacity plan is allocated. Supply a plan for steady-state or CUDA-graph use and warm up the JIT module once before capture.
- Returns:
topk_weightsandtopk_idshave shape(num_tokens, top_k).sorted_token_idsstores flattenedtoken * top_k + routeindices grouped by expert and padded with sentinelnum_tokens * top_k.expert_idsstores one expert perblock_mentries, andnum_tokens_post_paddedis a device-side int32 scalar naming the valid extent. Atomic scatter order within an expert is deliberately unspecified.- Return type:
Notes
When routed logits are exactly equal, the selected expert set and its order are unspecified; no lower-expert-ID tie break is guaranteed. The frozen CUDA device source of the original entries has SHA256
ec5bc689e68264a11a56a17fb10f699bc3733a521dea916b71ecda51d4227801; the routed 512-expert entries are generated separately (csrc/alphamoe_router/cake_*.cu). All are compiled with--use_fast_math. The operation has no dependency on an AlphaMoE compute kernel and its plan can feed either the W8A8 or NVFP4 fused up/down path.