Kimi K3-family MegaMoE¶
Sm100_KimiK3_MegaMoeConfig selects an isolated Kimi K3-family provider through
flashinfer.moe_ep.MoEEpMegaLayer. It supports SM100/SM103, 896 experts,
routed hidden size 3584, intermediate size 3072, top-16 routing, and EP sizes 1, 8,
or 16. Each rank owns a contiguous range of experts.
All EP ranks must share one NVLink domain. Across GB300 nodes, the container must expose the assigned IMEX channel so PyTorch can map peer memory through CUDA fabric handles. Two separate eight-GPU NVLink domains do not meet this requirement.
The provider accepts NVFP4 weights and activations (quant_kind="nvfp4")
or MXFP4 weights with MXFP8 activations (quant_kind="mxfp4_mxfp8").
Both paths produce BF16 output. combine_dtype selects BF16 or MXFP8
communication. SiTU and late routing weights are required.
Kimi K3 and K3.1 Pro¶
The published Kimi K3 config
and Kimi K3.1 Pro config
share this routed MoE contract. Both use the same config class and selector;
there is no model-name switch. Map fields from each checkpoint’s
text_config and retain its own weights, routes and calibration.
Checkpoint field |
API field |
Both models |
|---|---|---|
|
|
896 |
|
|
3584 |
|
|
3072 |
|
|
16 |
|
|
|
|
|
4.0 / 25.0 |
Use routed_expert_hidden_size, not the full model hidden_size=7168.
The API validates the routed dimensions and rejects a 7168-wide fleet.
The framework still owns routing, the 7168-to-3584 input projection, the
RMSNorm after routed MoE, the 3584-to-7168 output projection, and the two
shared experts. topk_weights must already include the model’s selected-score
normalization and routed scaling factor. Apply these weights only once,
after expert computation, as the kernel does by default.
The following application helper uses the existing API. bootstrap
identifies the EP group and local CUDA device. local_weights is a canonical
flashinfer.moe_ep.MoEWeightPack for this rank’s routed experts.
Checkpoint loading and packing remain the caller’s work.
from flashinfer import moe_ep
def make_routed_layer(text, local_weights, bootstrap, capacity,
*, quant_kind="mxfp4_mxfp8", **calibration):
config = moe_ep.Sm100_KimiK3_MegaMoeConfig(
intermediate_size=text["moe_intermediate_size"],
top_k=text["num_experts_per_token"],
activation=text["hidden_act"],
situ_beta=text["activation_situ_beta"],
situ_linear_beta=text["activation_situ_linear_beta"],
quant_kind=quant_kind,
combine_dtype="bf16",
**calibration,
)
return moe_ep.MoEEpMegaLayer(
bootstrap=bootstrap,
fleet_params=moe_ep.FleetParams(
num_experts=text["num_experts"],
token_hidden_size=text["routed_expert_hidden_size"],
max_tokens_per_rank=capacity,
),
weights=local_weights,
backend=moe_ep.MegaConfig(megakernel=config, quantize_input=True),
)
Both published checkpoints use MXFP4 routed weights; this example selects MXFP8 activations. The two model integrations differ only in the checkpoint config and weights they supply:
# Kimi K3
k3_layer = make_routed_layer(
k3_config["text_config"], k3_local_weights, bootstrap, capacity=256,
)
# Kimi K3.1 Pro
k31_layer = make_routed_layer(
k31_config["text_config"], k31_local_weights, bootstrap, capacity=256,
)
For either layer, pass the projected BF16 routed input and the model’s routes:
layer = k31_layer # Use k3_layer for Kimi K3.
workspace = layer.create_workspace(256)
tensors = moe_ep.MoEEpTensors(routed_hidden_states, topk_ids, topk_weights)
layer.warmup(tensors, workspace=workspace)
routed_output = layer(tensors, workspace=workspace) # [tokens, 3584]
For an NVFP4-converted checkpoint, set quant_kind="nvfp4" and pass its
input_norm_const, fc1_alpha, fc2_alpha and fc1_norm_const as
calibration arguments. Do not reuse another checkpoint’s calibration or
interpret MXFP4 packed bytes as NVFP4.
The shared contract is covered by CPU admission tests. The GPU tests below validate the routed kernel; they are not an end-to-end K3.1 Pro checkpoint accuracy or serving benchmark.
tests/moe_ep/test_sm100_kimi_k31.py is a runnable K3.1 Pro usage example.
It maps the checkpoint fields into the public API and checks eager execution
and CUDA graph replay with sparse weights and normalized route scores:
python -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k31.py
torchrun --standalone --nproc_per_node=8 -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k31.py
For EP16, use the multi-node launch below with this test file.
Weights and scales¶
Supply a flashinfer.moe_ep.MoEWeightPack in canonical
order: all FC1 gate rows, then all FC1 up rows. Let E be 896 / EP.
For prequantized weights, the packed E2M1 shapes are [E, 6144, 1792] for w13 and
[E, 3584, 1536] for w2, both torch.uint8. The low nibble holds
the even input element. Preprocessing applies the native gate/up layout
and scale swizzle.
Supply both unswizzled scale planes. NVFP4 uses E4M3 scales per 16 values:
[E, 6144, 224] and [E, 3584, 192]. MXFP4 uses E8M0 scales per 32
values: [E, 6144, 112] and [E, 3584, 96]. Scales can use their typed
FP8 dtype or raw torch.uint8 bytes. All tensors must be contiguous and
on the current CUDA device.
Unquantized BF16 or FP32 weights are also accepted: w13 has shape
[E, 6144, 3584] and w2 has shape [E, 3584, 3072]. Omit both
scale planes. Preprocessing quantizes each expert before execution.
For NVFP4, omitted FC1/FC2 alpha values use per-expert calibration from the
maximum absolute weight. Explicit alpha values instead set the global
weight scales through the equations below.
For NVFP4, pass the checkpoint calibration through input_norm_const
and FP32 [E] tensors fc1_alpha, fc2_alpha, and
fc1_norm_const. For global quantization scales gs_x, gs_w13,
gs_w2, and FC1 activation normalization nc, use:
input_norm_const = gs_x
fc1_alpha = 1.0 / (gs_x * gs_w13)
fc2_alpha = 1.0 / (nc * gs_w2)
fc1_norm_const = nc
Defaults of one are valid only for weights and inputs calibrated that way.
MoEEpTensors can override the three per-expert NVFP4 tensors for a call.
MXFP4/MXFP8 requires unit alpha and normalization values.
Inputs and activation¶
With quantize_input=True, supply BF16 [tokens, 3584] hidden states
and omit scales. With quantize_input=False, NVFP4 accepts packed
uint8 [tokens, 1792] plus E4M3 scales [tokens, 224]; MXFP8 accepts
float8_e4m3fn [tokens, 3584] plus E8M0 scales [tokens, 112].
Input scales use the same typed-or-byte convention as weight scales.
Input quantization follows the pinned Kimi kernels. In NVFP4, a nonzero
block whose scale rounds to zero saturates its nonzero payload values to
signed six; the zero scale makes the decoded block zero. MXFP8 uses the
upstream unit reciprocal for scale code zero. These rules differ from
quantizers that clamp all scales to the smallest normal value.
BF16 input needs 32-byte alignment. Packed input, input scales, expert IDs, and routing weights need 16-byte alignment.
Routing uses contiguous int32 or int64 [tokens, 16] expert IDs and
FP32 weights with the same shape. Active IDs must be distinct within each
token and have finite weights. Valid IDs are 0 through 895. An ID of
-1 masks a route; its weight is ignored, including NaN. Staging also
masks out-of-range IDs; it does not validate their values on the host.
Each rank may have a different
live token count, including zero, within the shared workspace capacity.
The SiTU gate is beta * tanh(gate / beta) * sigmoid(gate). The up branch
is linear_beta * tanh(up / linear_beta). Set both positive, finite
values from the model; neither has an implicit default. fast_situ=True enables the
source’s faster arithmetic and can change output bits.
Workspace and graph lifetime¶
Create and destroy workspaces collectively on all EP ranks in the same
order. Capacity must be positive and satisfy
capacity * 16 * 3584 < 2**31. knobs=None uses the native
heuristic; a dictionary pins native settings. Online tuning,
workspace pooling, and live-token tuning profiles are not supported.
MXFP4/MXFP8 uses asynchronous epilogue flags; both epi_flag_batches
values must be in [1, 4].
Warm each input shape before capture on every rank. CUDA graphs keep the
token shape fixed; update values in the captured input tensors before
replay. To change the live shape, use eager execution or capture another
graph. Keep tensor devices and dtypes fixed for each workspace.
num_valid_tokens_tensor is not supported.
By default, each call returns an owned output. Set MoEEpTensors.output
to a contiguous, 16-byte-aligned BF16 [tokens, 3584] CUDA tensor to reuse caller storage.
return_workspace_view=True borrows workspace output until the next
call on that workspace. It cannot be combined with caller output.
Synchronize CUDA work and release all graphs and borrowed views before
workspace.destroy() or layer.destroy(). Handles are independent;
destroying one does not release another. Destroying the layer releases
all its remaining handles.
Only FinalizeMode.DEFAULT and the inference kernel are supported.
In-kernel FC2 reduction, early routing weights, per-token FC1 scales,
persistent finalization, and unfinalized output are rejected.
Validation and timing¶
The original checks used B300 at EP1/EP8 and GB300 at EP16 (four nodes with four
GPUs each in one NVLink domain), with PyTorch
2.14.0a0+b2c75dd062.nv26.09 and CuTe DSL 4.8.0.
All four quantization/combine pairs passed the sparse FP64 layout oracle,
masked and empty inputs, changing eager counts, prequantized input, and
CUDA graph replay. The NVFP4/BF16 case also checks fast SiTU. Separate
tests compare exact quantizer bytes at zero, tiny values and midpoint ties,
and check masked reducer slots filled with NaNs. The sparse oracle checks
routing, gate/up order and scale layouts; it does not measure dense MMA
accumulation error.
The original seven distributed tests passed on every EP8 and EP16 rank, including
alternating eager calls and graph replays with the current standard provider
and separate caller buffers. The EP16 preflight also verified 16 distinct
GPU UUIDs, one fabric cluster/clique, and direct reads and writes to every peer.
The expanded eight-case suite in test_sm100_kimi_k3.py passed on B300
at EP1 and B200 at EP8. The added staged-output and flag-batch checks
have not been rerun at EP16.
Run the tests from the repository root:
python -m pytest --confcutdir=tests/cpu/moe_ep -q tests/cpu/moe_ep/test_kimi_k3_api.py tests/cpu/moe_ep/test_kimi_k3_packaging.py
python -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_kimi_k3_quantization.py
torchrun --standalone --nproc_per_node=8 -m pytest --confcutdir=tests/moe_ep -q tests/moe_ep/test_sm100_kimi_k3.py
For EP16 on four GB300 nodes with four GPUs each, launch the following once
per node. Set NODE_RANK to 0 through 3 and use the same reachable
MASTER_ADDR and MASTER_PORT on all four nodes:
export NVSHMEM_DISABLE_NVLS=1 NVSHMEM_MAX_TEAMS=512
torchrun --nnodes=4 --nproc_per_node=4 --node_rank="$NODE_RANK" \
--master_addr="$MASTER_ADDR" --master_port="$MASTER_PORT" \
-m pytest --confcutdir=tests/moe_ep -q \
tests/moe_ep/test_sm100_kimi_k3.py \
tests/moe_ep/test_sm100_kimi_k3_coexist.py
The EP16 coexistence run used NVSHMEM 3.7.2 with
NVSHMEM_DISABLE_NVLS=1 and NVSHMEM_MAX_TEAMS=512 on every rank.
Its default NVLS setup exhausted internal teams; increasing the team limit
alone did not complete setup during the retry. Disabling optional NVLS
collectives allowed the standard provider’s direct peer-memory path to run.
Kimi alone does not use NVSHMEM and passed without these settings.
Set FLASHINFER_KIMI_TEST_WHEEL to a built wheel to enable the two clean
installed-package import tests. The pinned export inputs and local changes
are listed in the drop’s VENDOR.md.
The public API benchmark includes staging, BF16 input quantization, compute
and combine into caller output. The following warm-graph medians are in
milliseconds, measured with deterministic sparse weights,
16 valid routes per token and fast_situ=False. Each result uses 30 samples
of ten graph replays; each sample uses the slowest EP rank. These adapter
measurements are separate from the earlier source-repository measurements.
Quantization |
Combine |
256 tokens/rank |
2048 tokens/rank |
8192 tokens/rank |
|---|---|---|---|---|
NVFP4 |
BF16 |
0.433 |
0.702 |
2.097 |
NVFP4 |
MXFP8 |
0.444 |
0.651 |
1.821 |
MXFP4/MXFP8 |
BF16 |
0.458 |
0.991 |
3.297 |
MXFP4/MXFP8 |
MXFP8 |
0.449 |
0.994 |
3.271 |
Three paired EP8 NVFP4/BF16 runs at 8192 tokens/rank measured current/Kimi
medians of 2.564/2.105, 2.558/2.111 and 2.560/2.098 ms.
The median paired speedup was 1.218x. The current standard selector
used the fixed tile (256, 128, 256); its default 8192-token tile failed
its own admission check. Both providers used the same supplied routes,
calibration, sparse weights, SiTU constants and default finalization, with
fast SiTU disabled. This comparison uses fixed knobs, without autotuning.
EP16 used 16 GB300 GPUs across four nodes in one NVLink domain, with the same benchmark settings. Each case also checked the first eight output rows on every rank against the sparse FP64 oracle. No current-provider timing comparison was run at EP16.
Quantization |
Combine |
256 tokens/rank |
2048 tokens/rank |
8192 tokens/rank |
|---|---|---|---|---|
NVFP4 |
BF16 |
0.232 |
0.609 |
2.102 |
NVFP4 |
MXFP8 |
0.241 |
0.500 |
1.627 |
MXFP4/MXFP8 |
BF16 |
0.249 |
0.793 |
2.743 |
MXFP4/MXFP8 |
MXFP8 |
0.244 |
0.737 |
2.614 |
The benchmark writes all samples, including each rank’s samples, to JSON.
torchrun --standalone --nproc_per_node=8 \
-m benchmarks.bench_kimi_k3_moe_ep --output kimi-public-times.json
torchrun --standalone --nproc_per_node=8 \
-m benchmarks.bench_kimi_k3_moe_ep --provider current \
--current-tile 256 128 256 --tokens 8192 --output current-public-times.json
- class flashinfer.moe_ep.Sm100_KimiK3_MegaMoeConfig(intermediate_size: int = 3072, top_k: int = 16, quant_kind: Literal['nvfp4', 'mxfp4_mxfp8'] = 'nvfp4', activation: Literal['situ'] = 'situ', situ_beta: float | None = None, situ_linear_beta: float | None = None, apply_routing_weights_before_fc2: bool = False, input_norm_const: float = 1.0, fc1_alpha: torch.Tensor | None = None, fc2_alpha: torch.Tensor | None = None, fc1_norm_const: torch.Tensor | None = None, knobs: dict | None = None, finalize_mode: FinalizeMode = FinalizeMode.DEFAULT, *, combine_dtype: Literal['bf16', 'mxfp8'] = 'bf16', fast_situ: bool = False, kernel_variant: Literal['inference'] = 'inference', enable_in_kernel_fc2_reduce: bool = False, enable_fc1_activation_per_token_scale: bool = False, num_valid_tokens_tensor: torch.Tensor | None = None)¶
K3/K3.1 Pro routed SiTU inference on SM100/SM103.
Uses 896 experts, routed width 3584, intermediate size 3072 and top-16. Read the routed width from
routed_expert_hidden_size, nothidden_size. Routing, latent projections, post-MoE norm and shared experts stay outside. Both SiTU beta values must be supplied, positive and finite.fast_situselects the source’s faster SiTU arithmetic; it can change output bits.knobsaccepts fixed native settings, not online tuning. NVFP4 alpha/norm tensors hold one FP32 value per local expert.