Kimi K3-family MegaMoE

Sm100_KimiK3_MegaMoeConfig selects an isolated Kimi K3-family provider through flashinfer.moe_ep.MoEEpMegaLayer. It supports SM100/SM103, 896 experts, routed hidden size 3584, intermediate size 3072, top-16 routing, and EP sizes 1, 8, or 16. Each rank owns a contiguous range of experts.

All EP ranks must share one NVLink domain. Across GB300 nodes, the container must expose the assigned IMEX channel so PyTorch can map peer memory through CUDA fabric handles. Two separate eight-GPU NVLink domains do not meet this requirement.

The provider accepts NVFP4 weights and activations (quant_kind="nvfp4") or MXFP4 weights with MXFP8 activations (quant_kind="mxfp4_mxfp8"). Both paths produce BF16 output. combine_dtype selects BF16 or MXFP8 communication. SiTU and late routing weights are required.

Kimi K3 and K3.1 Pro

The published Kimi K3 config and Kimi K3.1 Pro config share this routed MoE contract. Both use the same config class and selector; there is no model-name switch. Map fields from each checkpoint’s text_config and retain its own weights, routes and calibration.

Checkpoint fields used by the routed kernel

Checkpoint field

API field

Both models

num_experts

FleetParams.num_experts

896

routed_expert_hidden_size

FleetParams.token_hidden_size

3584

moe_intermediate_size

intermediate_size

3072

num_experts_per_token

top_k

16

hidden_act

activation

situ

activation_situ_beta / activation_situ_linear_beta

situ_beta / situ_linear_beta

4.0 / 25.0

Use routed_expert_hidden_size, not the full model hidden_size=7168. The API validates the routed dimensions and rejects a 7168-wide fleet. The framework still owns routing, the 7168-to-3584 input projection, the RMSNorm after routed MoE, the 3584-to-7168 output projection, and the two shared experts. topk_weights must already include the model’s selected-score normalization and routed scaling factor. Apply these weights only once, after expert computation, as the kernel does by default.

The following application helper uses the existing API. bootstrap identifies the EP group and local CUDA device. local_weights is a canonical flashinfer.moe_ep.MoEWeightPack for this rank’s routed experts. Checkpoint loading and packing remain the caller’s work.

from flashinfer import moe_ep

def make_routed_layer(text, local_weights, bootstrap, capacity,
                      *, quant_kind="mxfp4_mxfp8", **calibration):
    config = moe_ep.Sm100_KimiK3_MegaMoeConfig(
        intermediate_size=text["moe_intermediate_size"],
        top_k=text["num_experts_per_token"],
        activation=text["hidden_act"],
        situ_beta=text["activation_situ_beta"],
        situ_linear_beta=text["activation_situ_linear_beta"],
        quant_kind=quant_kind,
        combine_dtype="bf16",
        **calibration,
    )
    return moe_ep.MoEEpMegaLayer(
        bootstrap=bootstrap,
        fleet_params=moe_ep.FleetParams(
            num_experts=text["num_experts"],
            token_hidden_size=text["routed_expert_hidden_size"],
            max_tokens_per_rank=capacity,
        ),
        weights=local_weights,
        backend=moe_ep.MegaConfig(megakernel=config, quantize_input=True),
    )

Both published checkpoints use MXFP4 routed weights; this example selects MXFP8 activations. The two model integrations differ only in the checkpoint config and weights they supply:

# Kimi K3
k3_layer = make_routed_layer(
    k3_config["text_config"], k3_local_weights, bootstrap, capacity=256,
)

# Kimi K3.1 Pro
k31_layer = make_routed_layer(
    k31_config["text_config"], k31_local_weights, bootstrap, capacity=256,
)

For either layer, pass the projected BF16 routed input and the model’s routes:

layer = k31_layer  # Use k3_layer for Kimi K3.
workspace = layer.create_workspace(256)
tensors = moe_ep.MoEEpTensors(routed_hidden_states, topk_ids, topk_weights)
layer.warmup(tensors, workspace=workspace)
routed_output = layer(tensors, workspace=workspace)  # [tokens, 3584]

For an NVFP4-converted checkpoint, set quant_kind="nvfp4" and pass its input_norm_const, fc1_alpha, fc2_alpha and fc1_norm_const as calibration arguments. Do not reuse another checkpoint’s calibration or interpret MXFP4 packed bytes as NVFP4.

The shared contract is covered by CPU admission tests. The GPU tests below validate the routed kernel; they are not an end-to-end K3.1 Pro checkpoint accuracy or serving benchmark.

tests/moe_ep/test_sm100_kimi_k31.py is a runnable K3.1 Pro usage example. It maps the checkpoint fields into the public API and checks eager execution and CUDA graph replay with sparse weights and normalized route scores:

python -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k31.py
torchrun --standalone --nproc_per_node=8 -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k31.py

For EP16, use the multi-node launch below with this test file.

Weights and scales

Supply a flashinfer.moe_ep.MoEWeightPack in canonical order: all FC1 gate rows, then all FC1 up rows. Let E be 896 / EP. For prequantized weights, the packed E2M1 shapes are [E, 6144, 1792] for w13 and [E, 3584, 1536] for w2, both torch.uint8. The low nibble holds the even input element. Preprocessing applies the native gate/up layout and scale swizzle.

Supply both unswizzled scale planes. NVFP4 uses E4M3 scales per 16 values: [E, 6144, 224] and [E, 3584, 192]. MXFP4 uses E8M0 scales per 32 values: [E, 6144, 112] and [E, 3584, 96]. Scales can use their typed FP8 dtype or raw torch.uint8 bytes. All tensors must be contiguous and on the current CUDA device.

Unquantized BF16 or FP32 weights are also accepted: w13 has shape [E, 6144, 3584] and w2 has shape [E, 3584, 3072]. Omit both scale planes. Preprocessing quantizes each expert before execution. For NVFP4, omitted FC1/FC2 alpha values use per-expert calibration from the maximum absolute weight. Explicit alpha values instead set the global weight scales through the equations below.

For NVFP4, pass the checkpoint calibration through input_norm_const and FP32 [E] tensors fc1_alpha, fc2_alpha, and fc1_norm_const. For global quantization scales gs_x, gs_w13, gs_w2, and FC1 activation normalization nc, use:

input_norm_const = gs_x
fc1_alpha = 1.0 / (gs_x * gs_w13)
fc2_alpha = 1.0 / (nc * gs_w2)
fc1_norm_const = nc

Defaults of one are valid only for weights and inputs calibrated that way. MoEEpTensors can override the three per-expert NVFP4 tensors for a call. MXFP4/MXFP8 requires unit alpha and normalization values.

Inputs and activation

With quantize_input=True, supply BF16 [tokens, 3584] hidden states and omit scales. With quantize_input=False, NVFP4 accepts packed uint8 [tokens, 1792] plus E4M3 scales [tokens, 224]; MXFP8 accepts float8_e4m3fn [tokens, 3584] plus E8M0 scales [tokens, 112]. Input scales use the same typed-or-byte convention as weight scales. Input quantization follows the pinned Kimi kernels. In NVFP4, a nonzero block whose scale rounds to zero saturates its nonzero payload values to signed six; the zero scale makes the decoded block zero. MXFP8 uses the upstream unit reciprocal for scale code zero. These rules differ from quantizers that clamp all scales to the smallest normal value.

BF16 input needs 32-byte alignment. Packed input, input scales, expert IDs, and routing weights need 16-byte alignment.

Routing uses contiguous int32 or int64 [tokens, 16] expert IDs and FP32 weights with the same shape. Active IDs must be distinct within each token and have finite weights. Valid IDs are 0 through 895. An ID of -1 masks a route; its weight is ignored, including NaN. Staging also masks out-of-range IDs; it does not validate their values on the host. Each rank may have a different live token count, including zero, within the shared workspace capacity.

The SiTU gate is beta * tanh(gate / beta) * sigmoid(gate). The up branch is linear_beta * tanh(up / linear_beta). Set both positive, finite values from the model; neither has an implicit default. fast_situ=True enables the source’s faster arithmetic and can change output bits.

Workspace and graph lifetime

Create and destroy workspaces collectively on all EP ranks in the same order. Capacity must be positive and satisfy capacity * 16 * 3584 < 2**31. knobs=None uses the native heuristic; a dictionary pins native settings. Online tuning, workspace pooling, and live-token tuning profiles are not supported. MXFP4/MXFP8 uses asynchronous epilogue flags; both epi_flag_batches values must be in [1, 4].

Warm each input shape before capture on every rank. CUDA graphs keep the token shape fixed; update values in the captured input tensors before replay. To change the live shape, use eager execution or capture another graph. Keep tensor devices and dtypes fixed for each workspace. num_valid_tokens_tensor is not supported.

By default, each call returns an owned output. Set MoEEpTensors.output to a contiguous, 16-byte-aligned BF16 [tokens, 3584] CUDA tensor to reuse caller storage. return_workspace_view=True borrows workspace output until the next call on that workspace. It cannot be combined with caller output.

Synchronize CUDA work and release all graphs and borrowed views before workspace.destroy() or layer.destroy(). Handles are independent; destroying one does not release another. Destroying the layer releases all its remaining handles.

Only FinalizeMode.DEFAULT and the inference kernel are supported. In-kernel FC2 reduction, early routing weights, per-token FC1 scales, persistent finalization, and unfinalized output are rejected.

Validation and timing

The original checks used B300 at EP1/EP8 and GB300 at EP16 (four nodes with four GPUs each in one NVLink domain), with PyTorch 2.14.0a0+b2c75dd062.nv26.09 and CuTe DSL 4.8.0. All four quantization/combine pairs passed the sparse FP64 layout oracle, masked and empty inputs, changing eager counts, prequantized input, and CUDA graph replay. The NVFP4/BF16 case also checks fast SiTU. Separate tests compare exact quantizer bytes at zero, tiny values and midpoint ties, and check masked reducer slots filled with NaNs. The sparse oracle checks routing, gate/up order and scale layouts; it does not measure dense MMA accumulation error. The original seven distributed tests passed on every EP8 and EP16 rank, including alternating eager calls and graph replays with the current standard provider and separate caller buffers. The EP16 preflight also verified 16 distinct GPU UUIDs, one fabric cluster/clique, and direct reads and writes to every peer.

The expanded eight-case suite in test_sm100_kimi_k3.py passed on B300 at EP1 and B200 at EP8. The added staged-output and flag-batch checks have not been rerun at EP16.

Run the tests from the repository root:

python -m pytest --confcutdir=tests/cpu/moe_ep -q tests/cpu/moe_ep/test_kimi_k3_api.py tests/cpu/moe_ep/test_kimi_k3_packaging.py
python -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_kimi_k3_quantization.py
torchrun --standalone --nproc_per_node=8 -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k3.py

For EP16 on four GB300 nodes with four GPUs each, launch the following once per node. Set NODE_RANK to 0 through 3 and use the same reachable MASTER_ADDR and MASTER_PORT on all four nodes:

export NVSHMEM_DISABLE_NVLS=1 NVSHMEM_MAX_TEAMS=512
torchrun --nnodes=4 --nproc_per_node=4 --node_rank="$NODE_RANK" \
    --master_addr="$MASTER_ADDR" --master_port="$MASTER_PORT" \
    -m pytest --confcutdir=tests/moe_ep -q \
    tests/moe_ep/test_sm100_kimi_k3.py \
    tests/moe_ep/test_sm100_kimi_k3_coexist.py

The EP16 coexistence run used NVSHMEM 3.7.2 with NVSHMEM_DISABLE_NVLS=1 and NVSHMEM_MAX_TEAMS=512 on every rank. Its default NVLS setup exhausted internal teams; increasing the team limit alone did not complete setup during the retry. Disabling optional NVLS collectives allowed the standard provider’s direct peer-memory path to run. Kimi alone does not use NVSHMEM and passed without these settings.

Set FLASHINFER_KIMI_TEST_WHEEL to a built wheel to enable the two clean installed-package import tests. The pinned export inputs and local changes are listed in the drop’s VENDOR.md.

The public API benchmark includes staging, BF16 input quantization, compute and combine into caller output. The following warm-graph medians are in milliseconds, measured with deterministic sparse weights, 16 valid routes per token and fast_situ=False. Each result uses 30 samples of ten graph replays; each sample uses the slowest EP rank. These adapter measurements are separate from the earlier source-repository measurements.

B300 EP8 warm-graph forward time (ms)

Quantization

Combine

256 tokens/rank

2048 tokens/rank

8192 tokens/rank

NVFP4

BF16

0.433

0.702

2.097

NVFP4

MXFP8

0.444

0.651

1.821

MXFP4/MXFP8

BF16

0.458

0.991

3.297

MXFP4/MXFP8

MXFP8

0.449

0.994

3.271

Three paired EP8 NVFP4/BF16 runs at 8192 tokens/rank measured current/Kimi medians of 2.564/2.105, 2.558/2.111 and 2.560/2.098 ms. The median paired speedup was 1.218x. The current standard selector used the fixed tile (256, 128, 256); its default 8192-token tile failed its own admission check. Both providers used the same supplied routes, calibration, sparse weights, SiTU constants and default finalization, with fast SiTU disabled. This comparison uses fixed knobs, without autotuning.

EP16 used 16 GB300 GPUs across four nodes in one NVLink domain, with the same benchmark settings. Each case also checked the first eight output rows on every rank against the sparse FP64 oracle. No current-provider timing comparison was run at EP16.

GB300 EP16 warm-graph forward time (ms)

Quantization

Combine

256 tokens/rank

2048 tokens/rank

8192 tokens/rank

NVFP4

BF16

0.232

0.609

2.102

NVFP4

MXFP8

0.241

0.500

1.627

MXFP4/MXFP8

BF16

0.249

0.793

2.743

MXFP4/MXFP8

MXFP8

0.244

0.737

2.614

The benchmark writes all samples, including each rank’s samples, to JSON.

torchrun --standalone --nproc_per_node=8 \
    -m benchmarks.bench_kimi_k3_moe_ep --output kimi-public-times.json

torchrun --standalone --nproc_per_node=8 \
    -m benchmarks.bench_kimi_k3_moe_ep --provider current \
    --current-tile 256 128 256 --tokens 8192 --output current-public-times.json
class flashinfer.moe_ep.Sm100_KimiK3_MegaMoeConfig(intermediate_size: int = 3072, top_k: int = 16, quant_kind: Literal['nvfp4', 'mxfp4_mxfp8'] = 'nvfp4', activation: Literal['situ'] = 'situ', situ_beta: float | None = None, situ_linear_beta: float | None = None, apply_routing_weights_before_fc2: bool = False, input_norm_const: float = 1.0, fc1_alpha: torch.Tensor | None = None, fc2_alpha: torch.Tensor | None = None, fc1_norm_const: torch.Tensor | None = None, knobs: dict | None = None, finalize_mode: FinalizeMode = FinalizeMode.DEFAULT, *, combine_dtype: Literal['bf16', 'mxfp8'] = 'bf16', fast_situ: bool = False, kernel_variant: Literal['inference'] = 'inference', enable_in_kernel_fc2_reduce: bool = False, enable_fc1_activation_per_token_scale: bool = False, num_valid_tokens_tensor: torch.Tensor | None = None)

K3/K3.1 Pro routed SiTU inference on SM100/SM103.

Uses 896 experts, routed width 3584, intermediate size 3072 and top-16. Read the routed width from routed_expert_hidden_size, not hidden_size. Routing, latent projections, post-MoE norm and shared experts stay outside. Both SiTU beta values must be supplied, positive and finite. fast_situ selects the source’s faster SiTU arithmetic; it can change output bits. knobs accepts fixed native settings, not online tuning. NVFP4 alpha/norm tensors hold one FP32 value per local expert.