flashinfer.comm¶
This module provides communication primitives and utilities for distributed computing, including CUDA IPC, AllReduce operations, and memory management utilities.
CUDA IPC Utilities¶
|
|
|
Allocate a buffer and share it across the process group via CUDA IPC. |
|
Free a shared buffer previously created by |
DLPack Utilities¶
|
Pack a strided device allocation as a PyTorch tensor view. |
Mapping Utilities¶
|
A node with 8 GPUs, tp_size = 4, cp_size = 1, pp_size = 2 |
All-Gather Matmul¶
all_gather_matmul keeps its architecture-based default routing when
backend="auto". On SM100 and SM103, backend="cake" explicitly selects
the Cake fused backend for bfloat16 or float16 operands with K=8192: a
contiguous [M, 8192] input with any positive M, a [8192, N] weight
with N a positive multiple of 256 that is either contiguous or the
transposed view of a contiguous [N, 8192] parameter (each layout has its
own generated kernel; no copy is made), the process’s torch symmetric-memory
backend being the default CUDA backend or NVSHMEM (the Cake backend
never selects one itself), and a two-, four- or eight-rank NCCL process group.
The local input may be an ordinary contiguous CUDA tensor because Cake uses
internal symmetric scratch and flags for remote access and synchronization;
the scratch of a group grows to the largest M seen, and that growth is the
only collective after the first call. Unsupported explicit Cake requests raise
instead of silently falling back. One generated source per kernel serves both
architectures; the JIT loader compiles it with the exact flag set of the
device it runs on.
prepare_all_gather_matmul binds the weight, the process group and a row
capacity max_rows (default: the rows of the sample input) once, sizing the
symmetric scratch in that single collective, and returns a callable that
accepts any contiguous input with the same dtype, device and K and at
most max_rows rows. Both backend="auto" and backend="cake" select
this prepared Cake route; the same operand rules as the one-shot Cake route
apply. The returned launcher is CUDA-graph capturable: a captured call
allocates nothing but its output, and a call that would need a larger scratch
raises inside capture instead of running a collective. Unsupported
configurations raise during preparation instead of falling back.
|
Run push-wait all-gather matmul. |
|
Prepare a capacity-bound Cake all-gather matmul launcher. |
Fused FP8 Row-wise GEMM + All-Reduce¶
fp8_rowwise_gemm_allreduce computes, for an NCCL process group of two,
four or eight ranks on one node, Y = sum_r (A_r @ W_r.T) * (a_scale_r[:, None]
* w_scale_r[None, :]) with the per-rank FP8 (e4m3) operands A_r of shape
[M, K_l] and W_r of shape [N, K_l], fp32 row and column scales
applied before the cross-rank sum, and a bfloat16 result [M, N] on every
rank. backend="cake" (the only backend; "auto" selects it) runs one
fused kernel per call: the GEMM epilogue stores each owned tile’s bf16 partial
into this rank’s output and publishes every other tile’s partial into the
chunk owner’s symmetric receive slot, the owner sums the partials in fixed
rank order and broadcasts the result (through the NVLink multicast address on
the eight-rank Blackwell carrier), so no separate all-reduce kernel or
collective runs. The delivered architectures and world sizes are recorded in
the loader’s route tables (flashinfer.jit.cake_fp8_rowwise_gemm_allreduce);
a device or group outside them raises instead of falling back.
Operands are contiguous CUDA tensors starting at a 16-byte aligned address
(a, w, w_scale and a caller-provided out), with N a positive
multiple of 256 and at most 134184960, K_l a positive multiple of 128,
any positive M with M * N <= 2**30, and the process’s torch
symmetric-memory backend being the default CUDA backend or NVSHMEM (the Cake backend never selects one itself). The symmetric
workspace of a group grows to the largest M seen; that growth is the only
collective after the first call.
prepare_fp8_rowwise_gemm_allreduce binds the process group, N, K_l
and a row capacity max_rows (default: the rows of the sample input) once,
sizing the symmetric workspace in that single collective, and returns a
callable launcher(a, a_scale, w, w_scale, *, out=None) that accepts any
operands of the bound N and K_l with at most max_rows rows and
never runs another collective. The launcher is CUDA-graph capturable: a
captured call allocates nothing but its output and replays interleave with
eager calls; a call that exceeds the capacity raises before any launch.
max_rows must be identical on every rank of the group: the preparation
checks it in that collective and raises ValueError on every rank when the
values differ.
|
Fused row-wise FP8 GEMM + all-reduce over |
|
Bind |
TensorRT-LLM AllReduce¶
Types and Enums¶
Core Operations¶
|
Parameters: - allreduce_in: the input tensor. [token_num, hidden_dim] - world_size: the size of the process group. - world_rank: the rank of the current process. - token_num: the number of tokens in the sequence. - hidden_dim: the dimension of the hidden states. - workspace_ptrs: the workspace pointers. - launch_with_pdl: whether to launch with pdl. - use_oneshot: whether to use oneshot. If None, internal heuristics will be used. - trigger_completion_at_end: whether to trigger completion at the end. - fp32_acc: whether to use fp32 accumulation. - pattern_code: the pattern code. - allreduce_out: the output tensor. [token_num, hidden_dim] - residual_in: the residual input tensor. [token_num, hidden_dim] - residual_out: the residual output tensor. [token_num, hidden_dim] - norm_out: the norm output tensor. [token_num, hidden_dim] - quant_out: the quant output tensor. [token_num, hidden_dim] - scale_out: the scale output tensor. Initialization reference: tests/comm/test_trtllm_allreduce_fusion.py - rms_gamma: the rms gamma tensor. [hidden_dim] - rms_eps: the rms epsilon value. - scale_factor: the scale factor. For cudaGraphs safety, it should be a tensor. - layout_code: the layout code. - metadata: optional workspace metadata dict from create_ipc_workspace_for_all_reduce_fusion. If provided, it validates that world_size == tp_size, the FP32 Lamport mode matches the input dtype, and token_num * hidden_dim does not exceed max_token_num * metadata["hidden_dim"]. Raises ValueError if validation fails. - block_quant_group_size: group size (in elements along hidden_dim) for per-token-group block-wise FP8 quantization patterns (e.g. |
|
Parameters: - world_size: the size of the process group. - world_rank: the rank of the current process. - token_num: the number of tokens in the sequence. - hidden_dim: the dimension of the hidden states. - workspace_ptrs: the workspace pointers. - launch_with_pdl: whether to launch with pdl. - residual_in: the residual input tensor. [token_num, hidden_dim] - rms_gamma: the rms gamma tensor. [hidden_dim] - rms_eps: the rms epsilon value. - scale_factor: the scale factor. - moe_reduction_device_num_experts: the number of experts. - moe_reduction_scale_input: the scale input tensor. [token_num, hidden_dim] - moe_reduction_active_experts_token_input: the active experts token input tensor. [token_num, hidden_dim] - moe_reduction_token_input: the token input tensor. [token_num, hidden_dim] - layout_code: the layout code. - moe_allreduce_out: the moe allreduce output tensor. [token_num, hidden_dim] - residual_out: the residual output tensor. [token_num, hidden_dim] - norm_out: the norm output tensor. [token_num, hidden_dim] - quant_out: the quant output tensor. [token_num // 4, hidden_dim], fp16/bf16 -> fp4 - scale_out: the scale output tensor. Initialization reference: tests/comm/test_trtllm_moe_allreduce_fusion.py - weight_bias: bias added to rms_gamma before scaling. None or 0.0 -> standard RMSNorm (out = gamma * x * rsqrt(...)). 1.0 -> Gemma / Qwen3.5 RMSNorm (out = (1 + gamma) * x * rsqrt(...)). - backend: |
|
Parameters: - allreduce_in: the permuted/padded MoE expert output tensor. Shape [num_permuted_rows, hidden_dim]. Rows are referenced by expanded_idx_to_permuted_idx; num_permuted_rows may be larger than token_num * top_k due to expert padding. - residual_in: the residual input tensor. [token_num, hidden_dim] - norm_weight: the norm weight tensor. [hidden_dim] - expanded_idx_to_permuted_idx: the expanded index to permuted index tensor. [token_num, top_k] - norm_out: the norm output tensor. [token_num, hidden_dim] - residual_out: the residual output tensor. [token_num, hidden_dim] - quant_out: the packed FP4 output buffer, token_num * hidden_dim // 2 bytes (any element type; the Cake backend checks the byte size). - scale_out: the E4M3 scale output buffer in SWIZZLED_128x4 layout, round_up(token_num, 128) * round_up(hidden_dim // 16, 4) bytes. - workspace_ptrs: the workspace pointers. - launch_with_pdl: whether to launch with pdl. - world_rank: the rank of the current process. - world_size: the size of the process group. - eps: the epsilon value. - shared_expert_output: the shared expert output tensor. [token_num, hidden_dim] - expert_scale_factor: the expert scale factor tensor. [token_num, top_k] - routed_scaling_factor: the routed scaling factor. - weight_bias: bias added to rms_gamma before scaling. None or 0.0 -> standard RMSNorm (out = gamma * x * rsqrt(...)). 1.0 -> Gemma / Qwen3.5 RMSNorm (out = (1 + gamma) * x * rsqrt(...)). - backend: implementation to launch. |
Workspace Management¶
Parameters: - tp_rank: the rank of the current process. |
|
Release the symmetric-memory references held for an IPC workspace. |
|
Destroy a workspace created by trtllm_create_ipc_workspace_for_all_reduce_fusion. |
Initialization and Utilities¶
|
Initialize a single Lamport-style buffer to negative zero. |
|
Initialize three Lamport buffers to negative zero. |
Compute the padded size (rows times columns) of the FP4 swizzled layout. |
Unified AllReduce Fusion API¶
|
AllReduce + RMSNorm fusion operation, with optional FP8/NVFP4 quantization for supported backends. |
|
Create workspace for AllReduce fusion operations. |
|
Base class for AllReduce fusion workspaces. |
All-reduce workspaces backed by SymmDeviceMemory preserve their CUDA
virtual addresses across process checkpoint/restore. After quiescing all
work, release the physical handles and restore them with a fresh communication
backend before replaying a captured CUDA graph:
workspace.checkpoint_prepare()
workspace.checkpoint_restore(comm_backend)
Both methods are collective. Every rank must call them in the same order, and
comm_backend must reproduce the original rank and world size. Repeated
calls are no-ops after the workspace reaches the requested state. If an
exception occurs after detach or reattach begins, do not retry or reuse the
workspace; restart the affected rank. Workspaces backed by torch symmetric
memory do not support this lifecycle.
- class flashinfer.comm.TRTLLMAllReduceFusionWorkspace(tp_size: int, tp_rank: int, max_token_num: int, hidden_dim: int, dtype: dtype = torch.float16, comm_backend: CommBackend | None = None, group: ProcessGroup | None = None)¶
Bases:
AllReduceFusionWorkspaceTensorRT-LLM workspace for AllReduce fusion.
- __init__(tp_size: int, tp_rank: int, max_token_num: int, hidden_dim: int, dtype: dtype = torch.float16, comm_backend: CommBackend | None = None, group: ProcessGroup | None = None)¶
Create TensorRT-LLM AllReduce fusion workspace.
- Parameters:
tp_size – Tensor parallel size (world size)
tp_rank – Tensor parallel rank
max_token_num – Maximum number of tokens
hidden_dim – Hidden dimension size
dtype – Data type
comm_backend – Communication backend
group – Process group for symmetric memory rendezvous. Defaults to torch.distributed.group.WORLD.
- property backend: str¶
Return backend name.
- checkpoint_prepare() None¶
Detach physical backing; repeated successful calls are no-ops.
- checkpoint_restore(comm_backend: CommBackend) None¶
Restore physical backing; repeated successful calls are no-ops.
- Parameters:
comm_backend (CommBackend) – Communication backend used to recreate and exchange workspace memory handles. It must have the same rank and world size as the original allocation.
- destroy() None¶
Destroy workspace and free resources.
- class flashinfer.comm.MNNVLAllReduceFusionWorkspace(mapping: Mapping, max_num_tokens: int | None = None, hidden_dim: int | None = None, dtype: dtype | None = None, buffer_size_in_bytes: int | None = None, comm_backend: CommBackend | None = None)¶
Bases:
AllReduceFusionWorkspace- __init__(mapping: Mapping, max_num_tokens: int | None = None, hidden_dim: int | None = None, dtype: dtype | None = None, buffer_size_in_bytes: int | None = None, comm_backend: CommBackend | None = None)¶
Initialize the MNNVL Allreduce Fusion Workspace. The workspace will be allocated and initialized based on the provided problem size. If the maximum problem size in bytes exceeds the one-shot threshold, the workspace will be created according to the max of required one-shot size at the threshold or the required two-shot size. Note that the workspace is not bound to the given problem size. It can be reused for different problem sizes without reinitialization given the allocated size is sufficient.
If the buffer_size_in_bytes is provided, the workspace will be created according to the provided size. The user is expected to use the utility function get_required_buffer_size_bytes to calculate the required size. The actual allocation size may be larger due to alignment requirements. This covers the advanced use case, for example, the user may want to enforce oneshot strategy and ignore the heuristics.
Either max_num_tokens or buffer_size_in_bytes must be provided.
comm_backend will be used for creating the workspace and synchronization. If not provided, MPIBackend will be used which will use COMM_WORLD for synchronization.
- Parameters:
mapping – Mapping configuration containing rank info
max_num_tokens – The maximum number of tokens in the input tensor.
hidden_dim – The hidden dimension of the tensors to be reduced.
dtype – The data type of the tensors to be reduced.
buffer_size_in_bytes – The requested size in bytes for each lamport buffer. The actual allocation size may be larger due to alignment requirements. The actual usable size will be NUM_LAMPORT_BUFFERS * actual_buffer_size_per_lamport_buffer.
- property backend: str¶
Return backend name.
- checkpoint_prepare() None¶
Detach physical backing; repeated successful calls are no-ops.
- checkpoint_restore(comm_backend: CommBackend) None¶
Restore physical backing; repeated successful calls are no-ops.
- Parameters:
comm_backend (CommBackend) – Communication backend used to recreate and exchange MNNVL memory handles. It must have the same rank and world size as the original allocation.
- destroy() None¶
Destroy workspace and free resources.
- static get_required_buffer_size_bytes(tp_size: int, num_tokens: int, hidden_dim: int, dtype: dtype, strategy: MNNVLAllreduceFusionStrategy = MNNVLAllreduceFusionStrategy.AUTO) int¶
Calculate the required buffer size for a given problem size.
- is_buffer_size_sufficient(tp_size: int, num_tokens: int, hidden_dim: int, dtype: dtype, strategy: MNNVLAllreduceFusionStrategy = MNNVLAllreduceFusionStrategy.AUTO) bool¶
Calculate the required buffer size for a given problem size.
MNNVL CuTe DSL backend¶
create_allreduce_fusion_workspace(backend="mnnvl-cutedsl", ...) builds an
MNNVLCuteDSLAllReduceFusionWorkspace:
CuTe DSL kernels for the ranks of one NVLink multicast domain, compiled for
one static problem. backend="auto" never selects it. The keyword-only
arguments of the factory (top_k, rms_eps, weight_bias,
routed_scaling_factor, include_shared_expert, add_residual,
write_residual_output, apply_rms_norm, output_dtype,
dynamic_fp8, write_norm_output and config) are compile-time
settings: every allreduce_fusion() call on the workspace must use the
same epsilon, weight bias, residual, norm and output mode, and at most
max_token_num tokens of hidden_dim BF16 elements. Other backends
reject these arguments. residual_in must be identical on every rank, as
under tensor parallelism: most kernels add the residual once, on the rank that
owns a token or a hidden-dimension shard.
The kernels spin on data written by the other ranks and some of them occupy every SM. Do not run two of these collectives (for example from two workspaces) concurrently on different streams: a rank whose SMs are all held by one collective cannot start the other, which the peers may be waiting on.
The workspace picks a static profile for (world_size, hidden_dim, top_k,
dtype) and the device’s compute capability, then routes each token count to
a low-latency (LL), balanced (BT) or high-throughput (HT) kernel. Construction
fails when no profile matches; is_mnnvl_cutedsl_supported() runs the same
checks without raising. It describes the local rank only, and construction is
collective, so agree on the result across ranks first:
ok = is_mnnvl_cutedsl_supported(
4, 8192, torch.bfloat16, max_token_num=2048, apply_rms_norm=False
)
flag = torch.tensor(int(ok), device="cuda")
torch.distributed.all_reduce(flag, op=torch.distributed.ReduceOp.MIN)
if flag.item():
workspace = create_allreduce_fusion_workspace(
backend="mnnvl-cutedsl",
world_size=4,
rank=rank,
max_token_num=2048,
hidden_dim=8192,
dtype=torch.bfloat16,
apply_rms_norm=False,
add_residual=False,
)
out = allreduce_fusion(
input=x, workspace=workspace, pattern=AllReduceFusionPattern.kAllReduce
)
A workspace built with apply_rms_norm=False and add_residual=False
serves kAllReduce; the default norm-on workspace serves
kARResidualRMSNorm and the MoE finalize pattern. An
output_dtype=torch.float8_e4m3fn workspace serves the static FP8 patterns
(2 and 12, or 4 with write_norm_output=True); one that also sets
dynamic_fp8=True serves only the dynamic patterns (10, or 11 with
write_norm_output=True), not the static ones.
|
Whether this rank passes the |
- class flashinfer.comm.mnnvl_cutedsl_ar.MNNVLCuteDSLAllReduceFusionWorkspace(tp_size: int, tp_rank: int, max_token_num: int, hidden_dim: int, dtype: dtype, *, group: ProcessGroup | None = None, top_k: int = 10, rms_eps: float = 1e-06, routed_scaling_factor: float = 1.0, weight_bias: float = 0.0, include_shared_expert: bool = True, add_residual: bool = True, write_residual_output: bool = True, apply_rms_norm: bool = True, config: MNNVLCuteDSLConfig | None = None, output_dtype: dtype = torch.bfloat16, write_norm_output: bool = False, dynamic_fp8: bool = False)¶
Bases:
AllReduceFusionWorkspaceCompiled LL, BT, and HT protocols for one static problem shape.
Workspace construction is collective over
group, compiles the selected kernels and must finish before the first invocation. Calls using the same workspace must not overlap. Feature-disabled tensor slots use internal placeholders that are not read. Every setting below is fixed at construction:max_token_num,hidden_dim,top_k,rms_eps,weight_bias, the residual/norm/output flags and the output dtype, so a call must match them.flashinfer.comm.is_mnnvl_cutedsl_supported()runs the same argument and device checks without raising; it cannot see allocation or multicast failures during construction.residual_inmust be the same on every rank, as tensor parallelism replicates it: the BT and HT protocols and the LL two-shot all-reduce add the residual on the rank that owns a token or shard, while the LL one-shot adds each rank’s own copy, so differing residuals give protocol-dependent results.The kernels spin on data from the other ranks. Two such collectives that fill the GPU, for example two workspaces’ BT all-reduces at large token counts, must not run concurrently on different streams: a rank whose SMs are all held by one of them cannot start the other, which its peers may be waiting on.
configholds static profiles keyed by TP size, hidden size,top_k, dtype and optionally the GPU compute capability; the profile for the current device’s architecture wins over an architecture-agnostic one. Each profile routes every token count to an LL, BT or HT kernel preset. The default isDEFAULT_CONFIGwith the norm andNO_NORM_CONFIGwithout.Static per-tensor FP8 output: set
output_dtype=torch.float8_e4m3fnto servekARResidualRMSNormFP8QuantandkMoEFinalizeARResidualRMSNormFP8Quant, or additionallywrite_norm_output=Trueto servekARResidualRMSNormOutFP8Quant, which also writes the BF16 norm.Dynamic per-token FP8 output: additionally set
dynamic_fp8=Trueto servekARResidualRMSNormDynamicFP8Quant(or, withwrite_norm_output=True,kARResidualRMSNormOutDynamicFP8Quant). Each token getsscale_out[t] = max(amax(|norm_t|) / 448, 2**-9 / 448)andquant_out[t] = E4M3(norm_t * (1 / scale_out[t])), computed from the FP32 norm with the TRT-LLM MNNVL recipe;scale_outis a caller-provided[m, 1]FP32 tensor. The values match TRT-LLM up to FP32 rounding: the norm is(x * inv_rms) * gammarather than(gamma * x) * inv_rms, the quantizer multiplies by a per-token reciprocal rather than dividing, and a NaN stays NaN where TRT-LLM’s clamp makes it-448.Inputs, communication buffers, residual outputs and norm weights stay BF16. The output mode is fixed when the kernels compile, so a model that needs both BF16 and FP8 outputs uses one workspace per mode. Existing shape profiles are reused without retuning. FP8 output requires
apply_rms_norm=True.apply_rms_norm=Falsecompiles the collective without its normalisation: the chain stops after finalize, shared-expert add, all-reduce and the optional residual add, and writes that value toresidual_out. It then requireswrite_residual_output=True– otherwise nothing would be written – andrms_gammaandnorm_outmust beNone. The high-throughput protocol fuses the norm into a warp group and rejects it.Plain all-reduce: a workspace built with
apply_rms_norm=Falseandadd_residual=Falsealso serveskAllReduce, which writes the sum tooutput(allocated when omitted) and returns it.- __init__(tp_size: int, tp_rank: int, max_token_num: int, hidden_dim: int, dtype: dtype, *, group: ProcessGroup | None = None, top_k: int = 10, rms_eps: float = 1e-06, routed_scaling_factor: float = 1.0, weight_bias: float = 0.0, include_shared_expert: bool = True, add_residual: bool = True, write_residual_output: bool = True, apply_rms_norm: bool = True, config: MNNVLCuteDSLConfig | None = None, output_dtype: dtype = torch.bfloat16, write_norm_output: bool = False, dynamic_fp8: bool = False) None¶
Cake NVFP4-Transport AllReduce¶
cake_fp4_allreduce sums [rows, 6144] bfloat16 activations over a two-
or four-rank NCCL process group through NVFP4 records: every rank quantises
its rows (E2M1 values, one E4M3 scale per 16 values, one float32 row scale),
pushes them into Lamport receive windows of its peers and decodes the gathered
records in rank order with float32 accumulation, so the result is bitwise
identical on every rank. World size 2 runs a one-shot exchange, world size 4
a two-shot exchange whose owner ranks reduce, re-encode and all-gather (one of
three phase-B schedules is chosen per call from the rows each CTA owns).
cake_fp4_allreduce_nvfp4 accepts pre-quantised [rows, 3456] records
with one static scale per rank.
A CakeFp4AllReduceWorkspace is created collectively on every rank of the
group with the same family ("allreduce" or "rmsnorm"), input format
("bf16" or "nvfp4") and row capacity; it shares the receive windows
through CUDA IPC, arms their sentinels and publishes the peer window table.
Calls accept any row count up to the capacity (a multiple of 4 for world size
4), perform no allocation when the output buffers are given and no host
synchronisation, so they can be captured into CUDA graphs; the
launch_with_pdl flag issues the kernel as a programmatic dependent launch.
The programs are exact-architecture builds for SM100 (B200), SM103 (B300) and
SM107;
the build targets follow FLASHINFER_CUDA_ARCH_LIST when set, else the
visible devices.
cake_fp4_allreduce_rmsnorm (family "rmsnorm", world size 2) fuses the
exchange with the residual add and a Gemma RMSNorm: z = sum + residual in
float32, residual_out = bf16(z), rinv = rsqrt(mean(z^2) + eps) from the
float32 sum of squares and y = bf16(z) * rinv * (weight_bias + weight),
with eps and weight_bias as runtime scalars of the
CakeFp4AllReduceFusedNormSpec. The spec selects the output: bfloat16
y, or y quantised to NVFP4 (packed E2M1 values and 128x4-swizzled E4M3
block scales for a caller-supplied float32 global scale – the layout and
arithmetic of fp4_quantize) or to MXFP8 (E4M3 values and swizzled UE8M0
scales, as mxfp8_quantize). cake_fp4_allreduce_rmsnorm_nvfp4 takes
pre-quantised records with the same epilogue. Both return a
CakeFp4AllReduceFusedNormResult.
|
Per-rank state of the NVFP4-transport all-reduce, created collectively. |
Output precision of the fused all-reduce + residual + RMSNorm epilogue. |
|
|
Request of the fused epilogue: residual add, then RMSNorm scaled by |
|
Outputs of one fused call. |
|
Sum |
|
Sum pre-quantised NVFP4 records over the workspace's group. |
|
All-reduce |
|
Sum pre-quantised NVFP4 records, add |
FP8 Quantized AllReduce¶
|
FP8 quantized two-shot AllReduce with per-block scaling. |
vLLM AllReduce¶
|
Perform an out-of-place all-reduce via the vLLM custom kernel. |
|
Release the resources held by a vLLM custom all-reduce handle. |
|
Initialize the vLLM custom all-reduce backend. |
|
Register a peer's IPC-shared buffer with the local all-reduce handle. |
|
Register graph-capture buffers across the all-reduce world. |
Return IPC metadata for graph-capture buffers. |
|
Return the size of the vLLM all-reduce metadata structure in bytes. |
PCIe IPC AllReduce¶
Custom all-reduce for intra-node PCIe machines without NVLink. Admission is a capability check — world size, dtype, workspace capacity, and enough payload for every rank to own a share — and shapes it rejects fall back to the caller’s own collective.
import flashinfer.comm as comm
# Collective: every rank builds the workspace with identical arguments.
# Size max_numel to the real workload -- an oversized one costs latency.
ws = comm.PcieIpcAllReduceWorkspace(group=group, max_numel=128 * 6144)
for x in activations: # same shapes, same order, all ranks
if ws.supports(x):
y = ws.all_reduce(x) # out-of-place
else:
y = x.clone()
dist.all_reduce(y, group=group) # unsupported shape: fall back
ws.destroy() # collective; all ranks together
supports() is a pure function of shape and dtype, so every rank reaches the
same answer without agreeing on one at runtime. It says nothing about speed: a
supported shape runs a seed launch configuration — one crossover keyed on the
payload in bytes, no per-machine constants — and warns once per workspace until
tune() has measured the real one and persisted
it, since the crossovers depend on the fabric.
get_pcie_ipc_launch_config() exposes that seed for a
(world_size, numel, elem_size) triple, and returns None only for shapes
the kernels cannot run.
The kernels spin on peer flags with no timeout, so ranks that disagree on
shape, dtype or call order hang rather than raise. One workspace serves one
CUDA stream; use rebind_stream() after
ordering the two if a move is genuinely needed.
Two data planes¶
Above roughly a megabyte the workspace can move the payload with the copy engines instead of with SM load/store. The reason is not that the copy engine is faster per hop — in isolation the two are comparable — but that it keeps its throughput when every rank transfers at once, where SM peer traffic loses bandwidth to the concurrency. So the gap widens with world size, which is what makes a second plane worth having at these payload sizes.
It is a second protocol rather than a flag on the existing kernels. The SM path
encodes readiness in the payload — +0.0 means “not yet written” — which a
copy engine can neither produce nor observe, so the ring synchronises through
monotonic flags on side streams instead. Nothing about the SM kernels changes.
Both planes are ordinary tuner candidates, so nothing needs to be declared at
the call site and there is no notion of a prefill or decode “phase”: the
configuration follows the payload in bytes.
PcieIpcVariant gains COPY_ENGINE_RING, a flat neighbour ring, and
COPY_ENGINE_ISLAND, a 4+4 decomposition offered only at world size 8 and
only on rootcplx-noswitch, where a ring’s two socket-crossing hops would
otherwise set the pace. On that variant blocks carries the ring’s sub-chunk
depth rather than a grid size.
Warning
Tuning covers the batch sizes it is given, and shapes above the largest one
are served by whatever was measured there. Leave tune_batches unset and
the ladder is derived from max_numel, which covers everything the
workspace admits; pass it explicitly and the top of the range is yours to
get right. tune() warns when an explicit
list leaves a gap, and refuses to persist results measured below the
default sample counts, since a table that is quietly wrong is worse than
one that is missing.
|
Shared workspace for the PCIe IPC all-reduce. |
alias of |
|
alias of |
|
|
Launch configuration for one shape, or |
|
Probe this rank's GPU against other visible GPUs, recording failures. |
|
Probe every rank and agree on one profile. |
PCIe IPC AllGather and ReduceScatter¶
These standalone workspaces support BF16, FP16, and FP32 tensors at world sizes
2, 4, and 8. max_numel sizes one rank’s shard: the AllGather input or the
ReduceScatter output. Tensors must be contiguous, rank-2, 16-byte aligned, and
contain complete 16-byte packs.
Their Python implementation lives in flashinfer.comm.pcie_ipc_collectives;
the public entry points remain the flashinfer.comm exports shown below.
As with PcieIpcAllReduceWorkspace, construction, calls, and destruction
are collective. Every rank must issue the same sequence and configuration, and
one workspace belongs to one ordered CUDA stream. Keep the workspace alive
until every captured CUDA graph replay has completed.
AllGather copies opaque 16-byte packs. ReduceScatter accumulates in FP32 and converts to the requested output dtype; its TP8 schedule converts one four-rank partial before the final accumulation, so bitwise agreement with NCCL is not part of the contract.
The TP8 CopyEngine AllGather and topology ReduceScatter schedules require live
UUID/NVML evidence that logical ranks 0..3 and 4..7 form the expected
two-island placement. AllGather falls back to recursive doubling when that
proof is unavailable; TP8 ReduceScatter reports the input as unsupported.
import flashinfer.comm as comm
ag = comm.PcieIpcAllGatherWorkspace(group, max_local_rows * hidden, dtype=x.dtype)
rs = comm.PcieIpcReduceScatterWorkspace(group, max_local_rows * hidden, dtype=x.dtype)
gathered = ag.all_gather(x) # [world_size * local_rows, hidden]
shard = rs.reduce_scatter(reduced_input) # [local_rows, hidden]
ag.destroy()
rs.destroy()
launch_config() returns a conservative seed. tune([hidden, ...]) checks
every candidate against NCCL, minimizes the maximum latency across ranks, and
persists exact-shape results. The cache key includes the collective, world size,
topology/rank-placement fingerprint, workspace limits, dtype, and shape.
The seed thresholds are heuristics; support does not imply a performance win.
The variant names combine communication schedule, transport, and batching:
COPY_ENGINE uses recursive doubling, and ONE_PACK means one 16-byte
pack per thread in a grid-sized batch. See
PCIe IPC AG/RS schedules for the seven
variants, synchronization/traffic models, topology boundary, and measurement
provenance.
|
Workspace for out-of-place PCIe IPC all-gather. |
|
|
|
All-gather kernel selected across the FFI boundary. |
|
Return a deterministic all-gather launch seed. |
|
Workspace for SUM reduce-scatter over PCIe IPC. |
|
|
|
Reduce-scatter kernel selected across the FFI boundary. |
Return a deterministic reduce-scatter launch seed. |
Ulysses Context-Parallel All-to-All¶
Communication for Ulysses context parallelism over the 4-D layout
[B, S, H, D]. Two layout transforms are provided; a typical attention
layer makes four collective calls (q/k/v through scatter_heads, the
output through gather_heads):
scatter_heads:[B, S_local, H, D] -> [B, S_global, H_local, D]— each rank keeps head slice[rank * H_local, (rank+1) * H_local)of the full sequence;gather_heads:[B, S_global, H_local, D] -> [B, S_local, H, D]— the inverse, returning all heads of this rank’s sequence shard,
with H_local = H // world_size and S_global = S_local * world_size.
All backends produce bit-identical results.
A third entry point, exchange_chunks, takes [1, 1, world_size, chunk]
to the same shape: chunk r goes to peer r and lands in slot rank
there, which is dist.all_to_all_single over an already-packed payload. It
interleaves nothing, so it is for operands a preceding pack (a quantizer, say)
already laid out destination-major — the head-axis transforms would make such a
producer emit head-major and repack. On the "pcie" RDMA routes it is also
the cheaper descriptor: each peer’s bytes are contiguous on both ends, so the
NIC addresses them with the plain memory region and a linear offset instead of
an interleaved MKey, which is what exempts it from the 65535-byte stride limit
that bounds H*D*element_size for the other two.
Backend policy. UlyssesCommunicator selects its backend in the
constructor, strictly before any IPC allocation or JIT compilation:
|
behavior |
|---|---|
|
fused-transpose NVLink-P2P kernel when the group is a verified
single-node all-pairs NVLink mesh with a supported world size
(2/4/6/8); the experimental PCIe transport where NVLink is
unavailable and |
|
force the fused kernel; raises on every rank (before IPC/JIT for topology failures) when it cannot be used. |
|
force the experimental single-node PCIe transport at world size
1/2/4/8. Size 1 is an identity path and does not JIT-compile or
arm a native transport. Size 2 uses CUDA P2P. Sizes 4/8 prefer
an all-RDMA route (per-rank mlx5 to every peer) and fall
back to all-P2P; |
|
force |
Typical fallback reasons reported by .fallback_reason (all conservative —
anything unknown or unverifiable selects NCCL): unsupported world size (only
2/4/6/8 have fused-kernel instantiations; world_size == 1 is a no-copy
passthrough), ranks spanning multiple hosts, missing pair-wise P2P or NVLink
between any two concrete GPUs (verified per pair via NVML, not “some active
link”), duplicate or unknown physical GPU identity, a topology probe error,
inconsistent per-rank decisions, or a runtime NVLink initialization failure
after a positive topology decision.
Constraints. The constructor is always collective (all ranks together);
UlyssesCommunicator.close() is collective when the NVLink or PCIe backend
was armed — for the pure NCCL backend, world_size == 1, or an auto
fallback whose NVLink cleanup already completed, close is local and
idempotent. Rank-local failures inside backend initialization or a
collective close are exchanged as group outcomes so all ranks jointly
clean up and raise (or fall back) instead of deadlocking, and a failed
close may be retried. All ranks must request the same backend and
agree on max_bytes and dtype; each rank may bind a different CUDA
device (device accepts torch.device, str or an int ordinal,
e.g. cuda:rank). With world_size > 1 the NCCL and NVLink backends
(NVLink’s exchange_chunks runs all_to_all_single too) require
group to support CUDA all-to-all (an NCCL process group), checked at
construction before any resource is armed. Operands must be contiguous 4-D
CUDA tensors of the construction dtype (float16 / bfloat16 / float32
plus the PCIe types listed below) on the construction device, every dim positive,
and nbytes at most
max_bytes; the element count of any one operand must additionally stay
within the int32 kernel index range. scatter_heads requires
H % world_size == 0
and gather_heads requires S_global % world_size == 0. Collectives
run on the current CUDA stream; all ranks must issue the same call sequence
with consistent shapes, one collective in flight per communicator at a
time.
Destination-passing and workspace mode. Both collectives accept an
optional preallocated out= tensor. The NCCL backend additionally accepts
a reusable UlyssesWorkspace, which owns the packed send and receive
buffers and removes their per-call allocations. Omitting both keywords keeps
the original allocation path unchanged. The NVLink backend uses out= but
does not consume the workspace because its IPC staging allocation is owned by
the communicator.
Registered NVLink outputs. On the NVLink backend’s Cake route (SM100 /
SM103, nvlink_kernel == "cake"), UlyssesCommunicator.allocate_output()
is collective, as on the PCIe backend: every rank allocates max_bytes bytes
and maps every peer’s allocation over CUDA IPC. A scatter_heads or
gather_heads whose out= is such an output writes each rank’s blocks
straight into the destination ranks’ outputs, with no pass through the
staging buffer. All ranks pass their output of the same registration;
outputs stay registered until close, which frees them. Caller-owned
outputs keep the staged path, and the two may alternate on one communicator.
Call allocate_output for every geometry before CUDA graph capture.
PCIe backend (experimental). It additionally accepts the 1-, 2- and
4-byte element types float16 / bfloat16 / float32, int8 / uint8 and
float8_e4m3fn / float8_e5m2; its mlx5 routes — the eight-rank 4+4 NUMA
hybrid and the FLASHINFER_ULYSSES_PCIE_ROUTE=rdma all-RDMA route —
require batch size 1 and a head-row pitch H * D * element_size <= 65,535
bytes and refuse CUDA graph capture, while the all-P2P route takes any batch
size and can be captured when every output comes from
UlyssesCommunicator.allocate_output(). The JIT module links
libibverbs, libmlx5 and the CUDA driver library even when topology selects
the all-P2P route, so the rdma-core development packages must be present;
it is the only FlashInfer JIT module that links them. Multi-rank PCIe calls
require out= from UlyssesCommunicator.allocate_output(); the
backend has no staging buffers, so workspace=,
UlyssesCommunicator.create_workspace() and the head-chunk primitives
are refused at world size above one. A rejection before native code runs
(a bad operand, a refused capture, a second stream) leaves the RDMA routes
usable, since every rank rejects the same call under the SPMD contract; the
all-P2P route, whose barrier has no abort protocol, treats it as fail-stop.
A failure inside the transport is fail-stop on every route: the communicator
enters a BROKEN state that permits only a collective close. On the RDMA
routes it publishes a sticky abort that releases the peers and close
completes; on all-P2P close refuses to synchronize and the process must
exit.
See the PCIe Ulysses design note
for the support matrix, output-lifetime rules, teardown protocol, routing
controls, and benchmark.
Known limitations.
PyTorch builds without
torch.cuda.get_device_properties(...).uuidcannot establish physical GPU identity:autoconservatively falls back to NCCL (the reason names the missing attribute).When each process can only see its own GPU (e.g. one
CUDA_VISIBLE_DEVICESentry per rank), peers are invisible to the P2P probe andautofalls back to NCCL.Out-of-range CUDA ordinals passed as strings or ints are rejected at construction; a pre-built
torch.deviceobject wraps its index into a signed byte before FlashInfer can see it (torch.device("cuda:256")is alreadycuda:0), so only the surviving index can be range-checked.Teardown metadata exchanges run bound to the communicator device; an extreme failure in the guard restore path after a completed collective can still desynchronize ranks (never observed in tests; tracked as a hardening note).
Example (Wan2.1-style attention; see the wan example for the full integration):
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)
with UlyssesCommunicator(group, max_bytes=q.nbytes,
dtype=torch.bfloat16, device=device) as comm:
q_ = comm.scatter_heads(q) # [B,S_local,H,D] -> [B,S_global,H_local,D]
k_ = comm.scatter_heads(k)
v_ = comm.scatter_heads(v)
o_ = attention(q_, k_, v_)
o = comm.gather_heads(o_) # [B,S_global,H_local,D] -> [B,S_local,H,D]
The v0.7.2 spelling max_elems=... (a capacity in elements of dtype)
is still accepted and converted to max_elems * dtype.itemsize bytes with a
DeprecationWarning; it is deprecated since v0.7.3 and scheduled for
removal in a future release. Pass exactly one capacity argument. comm.max_elems,
the v0.7.2 attribute, remains available in the construction dtype;
comm.max_bytes is the byte capacity used by the packed-chunk API.
Preallocated outputs and NCCL staging can be reused across serialized calls:
workspace = comm.create_workspace()
q_global = torch.empty(B, S_global, H_local, D,
dtype=q.dtype, device=q.device)
comm.scatter_heads(q, out=q_global, workspace=workspace)
Packed chunk exchange. exchange_chunks follows the byte-oriented
communicator API: it accepts contiguous [1, 1, P, C] tensors, sends chunk
r to rank r, and receives chunks in source-rank order. A packed Lowp
payload uses dtype=torch.uint8; this describes its existing encoding and
does not quantize or cast it. The NCCL and NVLink backends use NCCL for
this operation; the fused NVLink kernel applies only to ordinary head-layout
transforms. The experimental PCIe backend retains its registered-output path.
A one-rank exchange without an explicit output is an identity; explicit outputs
must not overlap the input.
For chunk exchange on NCCL or NVLink, allocate_output(x, "exchange_chunks",
dtype=...) creates ordinary PyTorch receive storage. PCIe chunk outputs and
NVLink Cake direct head-transform outputs follow the registration and lifetime
rules described above. create_workspace(max_elems=...) and
UlyssesWorkspace retain their element-based staging capacities:
# send_u8: destination-major, contiguous uint8 [P, chunk_bytes].
chunks = send_u8.view(1, 1, P, -1)
received = comm.allocate_output(chunks, "exchange_chunks", dtype=torch.uint8)
comm.exchange_chunks(chunks, out=received, dtype=torch.uint8)
- class flashinfer.comm.UlyssesCommunicator(group: ProcessGroup | None = None, *, max_bytes: int | None = None, dtype: dtype, backend: str = 'auto', nvlink_kernel: str = 'auto', device: device | str | int | None = None, max_elems: int | None = None)¶
Bases:
objectUlysses context-parallelism all-to-all communicator.
Provides the two layout transforms of Ulysses attention over the 4-D layout
[B, S, H, D](a typical attention layer makes four collective calls: q/k/v throughscatter_heads(), the output throughgather_heads()):scatter_heads():[B, S_local, H, D] -> [B, S_global, H_local, D](each rank keeps a head slice of the full sequence)gather_heads():[B, S_global, H_local, D] -> [B, S_local, H, D](each rank gets all heads of its local sequence shard back)
where
His the global head count,H_local = H // world_sizeandS_global = S_local * world_size. All backends produce bit-identical results.exchange_chunks()exchanges already-packed destination-major chunks. The NCCL and NVLink backends use NCCL for this operation.prepare_qkv()andscatter_qkv()additionally provide Sage2 INT8/FP8 QKV preparation, using this chunk exchange and a statistics AllGather on the same NCCL group.Backend selection for ordinary scatter/gather happens in the constructor, strictly before any IPC allocation or JIT compilation (see
resolve_ulysses_backend()):backend="auto": the fused-transpose NVLink-P2P kernel when the group is a verified single-node all-pairs NVLink mesh with a supported world size (2/4/6/8); where NVLink is unavailable andFLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1is set on every rank, the experimental PCIe transport below (it warns once); NCCL otherwise — including when NVLink runtime initialization fails after a positive topology decision. Inspectbackendandfallback_reasonfor the outcome.backend="nvlink": force the fused kernel; raises on every rank (before any IPC/JIT for topology failures) when it cannot be used.backend="pcie": explicitly enable the experimental single-node PCIe transport at world size 1/2/4/8. One rank is an identity path, two ranks use CUDA P2P, and four/eight ranks prefer an all-RDMA route (every peer’s payload over the rank’s mlx5 QP) with all-P2P fallback;FLASHINFER_ULYSSES_PCIE_ROUTE(auto/p2p/rdma/hybrid) forces all-P2P, all-RDMA at any multi-rank world size, or the eight-rank 4+4 NUMA hybrid (same-NUMA CUDA P2P plus cross-NUMA mlx5). Multi-rank outputs are registered up front withallocate_output()(there are no staging workspaces);"auto"reaches this backend only behind the opt-in above.backend="nccl": force thedist.all_to_all_singlepath; skips the topology/NVML probe and all IPC/JIT entirely (the constructor still resolves and guards the CUDA device and performs CUDA-backed metadata collectives overgroup). Supports any world size.
nvlink_kernel="portable"forces the portable CUDA kernel on NVLink;"cake"requires an available Cake export. All ranks must request the same backend and NVLink kernel policy. Withworld_size > 1the NCCL and NVLink backends requiregroupto support CUDA all-to-all (an NCCL process group); this is checked at construction before any resource is armed.world_size == 1is a passthrough: both collectives return the input tensor unchanged (no copy).The NVLink backend launches one of two kernels on the same IPC staging and signal allocations, decided once per communicator inside the NVLink initialization (every rank takes the same route;
nvlink_kernelreports it): the exported Cake programs offlashinfer.cake_ulysses_backendwhen the device’s compute capability is one that package delivers for this dtype and world size (SM100 / SM103), else FlashInfer’s portable NVLink kernel. On the Cake route, a collective whoseoutwas registered withallocate_output()writes every rank’s blocks straight into the peers’ registered outputs instead of passing them through the staging buffer; the two Cake kernels use disjoint words of one signal allocation and may alternate call by call. The raw entry points (init_ulysses_a2a/ulysses_a2a/dispose_ulysses_a2a) always drive the portable kernel; a communicator’s own collectives never mix the portable and the Cake kernels on its signal words.Constraints¶
The constructor is always collective: every rank of
groupmust call it together.close()is collective when the NVLink or PCIe backend was armed (their resources are peer-shared); for the pure NCCL backend,world_size == 1, or an auto fallback whose NVLink cleanup already completed,closeis local and idempotent. Rank-local failures inside the constructor’s NVLink initialization or inside a collectivecloseare exchanged as group outcomes, so all ranks jointly clean up and raise (or fall back) instead of deadlocking; a failedclosemay be retried by all ranks.Collectives run on the current CUDA stream of this rank; every rank must issue the same sequence of calls with consistently-shaped operands (a shape or call-order mismatch across ranks is a collective failure: expect hangs or garbage, exactly as with any collective library). At most one collective may be in flight per communicator at a time (the NVLink and PCIe signal protocols assume serialized calls); do not call one communicator concurrently from multiple streams or threads. The experimental PCIe P2P route enqueues asynchronously and is CUDA Graph capturable when every output comes from
allocate_output()(which is itself collective and not capturable); the hybrid and all-RDMA routes block on the host and refuse capture. PCIe collectives are bound to the stream of their first call.Operand tensors must be contiguous 4-D CUDA tensors of the construction
dtype(float16 / bfloat16 / float32) on the construction device, with every dim positive, at most2**31 - 1elements and at mostmax_bytesbytes.exchange_chunks()also accepts packed uint8 with a per-calldtypeoverride;scatter_heads()additionally requiresH % world_size == 0andgather_heads()requiresS_global % world_size == 0. PCIe additionally accepts int8/uint8 and the float8 storage types (any 1-, 2- or 4-byte element); its mlx5 routes (hybrid and all-RDMA) additionally require batch size 1 andH * D * element_size <= 65_535bytes, while the all-P2P routes take any batch size.Multi-rank PCIe calls require an explicit
outreturned byallocate_output(). This keeps registration lifetime and overwrite points explicit; pre-register one output per live result and geometry.A rejection before native code runs (a bad operand, a refused capture, a second stream) is decided from the call’s own arguments, so under the SPMD contract every rank rejects it: the RDMA routes raise the ordinary error and stay usable, while the all-P2P route, whose barrier has no abort protocol, poisons teardown and enters BROKEN. A failure inside the transport is fail-stop on every route: registered buffers, queue pairs and the GPU epoch counters may already be half-mutated, so the communicator enters BROKEN and permits only a collective
close(). On the RDMA routes it publishes a sticky abort that releases the peers andclose()completes; on all-P2Pclose()refuses to synchronize and reports that the process must exit.Each rank may use a different CUDA device (e.g.
cuda:rank); ranks must agree onmax_bytes,dtypeandbackend.
- param group:
Process group of the Ulysses ranks. Defaults to
dist.group.WORLD.- type group:
torch.distributed.ProcessGroup, optional
- param max_bytes:
Per-rank byte capacity of one communication operand, shared across dtypes and transforms. Sizes the NVLink staging buffer once at construction. For quantized QKV this must cover the complete packed Q/K/V payload, including scales and alignment, rather than one floating input. Statistics and other workspace allocations are separate; this is not a total memory budget. The maximum is
(2**31 - 1) * 4bytes, and each call separately enforces the int32 element-index limit.- type max_bytes:
int, optional
- param max_elems:
Deprecated since v0.7.3, scheduled for removal in a future release: the v0.7.2 capacity in elements of
dtype, converted tomax_elems * dtype.itemsizebytes with aDeprecationWarning. Pass exactly one ofmax_bytesandmax_elems.- type max_elems:
int, optional
- param dtype:
Default element type of operands (float16 / bfloat16 / float32; PCIe additionally int8 / uint8 / float8_e4m3fn / float8_e5m2); enforced on every call that does not name its own.
- type dtype:
torch.dtype
- param backend:
"auto"|"nvlink"|"pcie"|"nccl"(see above).- type backend:
str
- param nvlink_kernel:
"auto"|"cake"|"portable"; relevant when NVLink is used."portable"names the in-tree CUDA kernel’s architecture coverage, not a slow path: it uses the optimized vector mover when eligible.- type nvlink_kernel:
str
- param device:
CUDA device of this rank; normalized to an explicit index (bare
"cuda"means the current device, an int is a CUDA ordinal). Defaults to the current CUDA device.- type device:
torch.device or str or int, optional
Examples
>>> with UlyssesCommunicator(group, max_bytes=q.nbytes, dtype=torch.bfloat16) as comm: ... q_ = comm.scatter_heads(q) # [B,S_local,H,D] -> [B,S_global,H_local,D] ... ... ... o = comm.gather_heads(o_) # [B,S_global,H_local,D] -> [B,S_local,H,D]
- __init__(group: ProcessGroup | None = None, *, max_bytes: int | None = None, dtype: dtype, backend: str = 'auto', nvlink_kernel: str = 'auto', device: device | str | int | None = None, max_elems: int | None = None)¶
Construct a Ulysses communicator.
- Parameters:
group (Optional[ProcessGroup], optional) – Process group spanning the participating ranks.
Noneusestorch.distributed.group.WORLD.max_bytes (int, optional) – Per-rank upper bound in bytes on one communication operand. Quantized QKV requires the complete packed payload to fit. Used to size the backend workspace.
max_elems (int, optional) – Deprecated since v0.7.3 (scheduled for removal in a future release): the v0.7.2 form of the capacity, in elements of
dtype; converted withdtype.itemsize. Mutually exclusive withmax_bytes.dtype (torch.dtype) – Default element dtype for collective operands. Must be one of
torch.float16,torch.bfloat16, ortorch.float32;backend="pcie"additionally acceptstorch.int8,torch.uint8,torch.float8_e4m3fnandtorch.float8_e5m2.backend (str, default = "auto") – Backend selection policy.
"auto"probes topology and prefers NVLink when supported, then the experimental PCIe transport where NVLink is unavailable andFLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1is set on every rank, otherwise falls back to NCCL."nvlink"forces the NVLink backend and raises if unavailable."pcie"explicitly enables the experimental single-node 1/2/4/8-rank PCIe transport."nccl"forces the NCCL path.nvlink_kernel (str, default = "auto") –
"auto"uses Cake when an export exists, otherwise portable."portable"forces the portable NVLink kernel;"cake"requires a Cake export. Portable includes the optimized vector mover when eligible. All ranks must select the same value.device (Optional[Union[torch.device, str, int]], optional) – CUDA device bound to this rank.
Noneuses the current CUDA device. Strings and integers are normalized to an explicit CUDA ordinal.
- allocate_output(x: Tensor, op: str, *, dtype: dtype | None = None) Tensor¶
Allocate a registered output for one Ulysses layout transform.
For
backend="pcie"this is a collective cold-path operation: the fixed output uses native CUDA allocation, is registered with the selected transport, and remains registered untilclose(). This gives the caller a handle it owns for the communicator’s lifetime. Multi-rank PCIe calls require such an output; allocate every geometry during setup and pass it throughout=on the exchange.For
backend="nvlink"on the exported Cake route (SM100 / SM103,nvlink_kernel"cake") the layout transforms register the output the same way: every rank allocatesmax_bytesbytes with native CUDA allocation and maps every peer’s allocation over CUDA IPC. Ascatter_heads()orgather_heads()whoseout=is such an output then writes every rank’s blocks straight into the destination ranks’ outputs, with no pass through the staging buffer. Every rank must pass its output of the same registration (the same position in the sequence ofallocate_outputcalls). The memory belongs to the communicator and is freed byclose(); the returned tensor must not be used afterwards. Any other output, and every output of the portable NVLink kernel, keeps the staged path.- Parameters:
x (torch.Tensor) – Operand the output is sized for: a contiguous 4-D CUDA tensor with the shape
opconsumes.op (str) –
"scatter_heads","gather_heads"or"exchange_chunks". Outputs are registered per transform and are not interchangeable between them.dtype (torch.dtype, optional) – Element type for the calls this output serves, overriding the communicator dtype. For
exchange_chunks, NCCL/NVLink also accept float16, bfloat16, float32 and uint8; other overrides require PCIe. The allocation keeps the communicator’s byte budget, so a narrower dtype holds proportionally more elements.
- Returns:
Tensor with the shape
opproduces fromx, on the same device and dtype asx. On multi-rank pcie, and on the multi-rank NVLink Cake route forscatter_headsandgather_heads, it stays registered untilclose().- Return type:
torch.Tensor
- close() None¶
Release the communicator. Idempotent once fully closed.
Collective when the NVLink or PCIe backend was armed: every rank must call
closetogether. PCIe first requires every rank to report that native GPU/RDMA work is bounded; if any rank cannot prove that, no rank enters device synchronization or releases a registration, and process termination is required. A safe PCIe close then synchronizes, closes every output’s peer imports, disposes its MR/MKey registration, and finally disposes the transport. NVLink similarly synchronizes before releasing its kernel handle, peer mappings, and exports. Each stage has bounded group-coordinated retries. If a retryable teardown stage still cannot complete, every rank raises the same error and remains CLOSING; every rank may retryclose(). CLOSED is reached only after complete group-wide teardown. Pure NCCL holds no resources and closes locally.
- create_workspace(*, max_elems: int | None = None) UlyssesWorkspace¶
Allocate reusable NCCL send/receive staging buffers.
This operation is rank-local and non-collective.
max_elemsdefaults to the communicator capacity and may be smaller when a caller knows the maximum chunk size it will communicate. The multi-rank pcie backend has no staging buffers and rejects this call; its outputs are registered withallocate_output()instead.- Parameters:
max_elems (int, optional) – Capacity of each send and receive staging buffer in elements.
Nonefills the communicator’smax_bytescapacity at the communicator dtype.- Returns:
A workspace on the communicator’s device with its dtype.
- Return type:
- exchange_chunks(x: Tensor, *, out: Tensor | None = None, dtype: dtype | None = None) Tensor¶
[1, 1, W, C] -> [1, 1, W, C]: equal-length chunk all-to-all.Chunk
rof this rank’s input goes to peerr, and lands in slotrankof that peer’s output – the semantics oftorch.distributed.all_to_all_singleon an already-packed payload.This is the transform for a payload that was produced destination-major by a preceding pack (a quantizer, say), so its per-peer bytes are already contiguous.
scatter_heads()andgather_heads()do the head-axis interleave instead, which forces a head-major layout on the producer; this entry point removes that constraint. On the RDMA routes it is also the cheaper descriptor – a single-row MKey, exempt from the 65535-byte interleaved-stride limit that boundsH*D* element_sizefor the other two.- Parameters:
x (torch.Tensor) – Contiguous 4-D CUDA tensor shaped
[1, 1, world_size, chunk].chunkis in elements ofx’s dtype; a packed uint8 payload passesdtype=torch.uint8andchunkin bytes.out (torch.Tensor, optional) – Preallocated output of the same shape, not overlapping
x. Multi-rank PCIe requires an output fromallocate_output().dtype (torch.dtype, optional) – Element type for this call, overriding the communicator dtype. PCIe accepts its supported storage types; NCCL/NVLink accept float16, bfloat16, float32 and uint8. No conversion is performed.
- Returns:
Tensor of the same shape, device and dtype as
x.- Return type:
torch.Tensor
- gather_heads(x: Tensor, *, out: Tensor | None = None, dtype: dtype | None = None, workspace: UlyssesWorkspace | None = None) Tensor¶
[B, S_global, H_local, D] -> [B, S_local, H, D].Inverse of
scatter_heads(): gather all head slices for this rank’s local sequence shard. Runs on the current CUDA stream. Returns the input unchanged whenworld_size == 1.- Parameters:
x (torch.Tensor) – Contiguous 4-D CUDA tensor with shape
[B, S_global, H_local, D].out (torch.Tensor, optional) – Preallocated contiguous output with shape
[B, S_global // world_size, H_local * world_size, D]. Supplying it removes the public output allocation. It must not aliasxwhenworld_size > 1. Multi-rank PCIe requires an output returned byallocate_output(); on the NVLink Cake route such an output takes the direct path.dtype (torch.dtype, optional) – Element type for this call, overriding the communicator dtype. pcie backend only.
max_bytesis denominated in bytes, so a narrower dtype here simply fits more elements in the same capacity.workspace (UlyssesWorkspace, optional) – Reusable NCCL pack/receive storage. Supplying it removes the two NCCL staging allocations. It is validated but unused by the NVLink backend, and rejected by the multi-rank pcie backend, which has no staging buffers.
- Returns:
Tensor with shape
[B, S_local, H, D]on the same device and dtype asx.- Return type:
torch.Tensor
- gather_output_head_chunk(x: Tensor, *, local_heads: int, head_offset: int, out: Tensor, workspace: UlyssesWorkspace | None = None) Tensor¶
Gather one attention-output head band into a full destination.
xhas shape[B, S_global, head_count, D].outis the preallocated full output[B, S_local, world_size * local_heads, D]. Only the selected head band is written; calling this method for every non-overlapping band in a complete schedule reconstructs ordinary whole-head Ulysses output communication.The NCCL path packs the sequence split directly into reusable staging and merges the received head band directly into
out. The NVLink path uses its fused-transpose collective followed by the same merge primitive. Runs on the current CUDA stream.- Parameters:
x (torch.Tensor) – Positive-strided CUDA tensor with shape
[B, S_global, head_count, D].local_heads (int) – Total number of attention-output heads owned by this rank before the output all-to-all.
head_offset (int) – Start of this band within the rank’s
local_heads.out (torch.Tensor) – Preallocated contiguous destination with shape
[B, S_global // world_size, world_size * local_heads, D]. Only this head band is modified.workspace (UlyssesWorkspace, optional) – Reusable staging storage large enough for
x. A workspace cannot service concurrent collectives.
- Returns:
outafter merging the gathered head band.- Return type:
torch.Tensor
- input_buffer(out: Tensor, shape: Sequence[int]) Tensor¶
The transport’s own input staging buffer behind a registered output.
The RDMA routes never let the NIC read caller memory: every operand is copied into a landing buffer registered once alongside
out. Building the operand directly in the buffer returned here removes that copy – for a layer-scale scatter it is the largest device-to-device move on the path – and the caller’s own source allocation with it.The result must reach
scatter_heads()orgather_heads()unmodified: the fast path is exact pointer identity, and a slice or a re-view that moves the base pointer is rejected rather than silently staged, because on the scatter path the NIC reads the landing buffer whatever the caller passes.- Parameters:
out (torch.Tensor) – An output returned by
allocate_output().shape (Sequence[int]) – The
[B, S, H, D]operand shape to view the buffer as.
- Returns:
A view of transport-owned memory in the dtype
outwas registered under. It stays alive as long asoutdoes, and every exchange onoutoverwrites it – it is where the next operand is built, not where one is kept.- Return type:
torch.Tensor
- property max_elems: int¶
Legacy capacity in elements of the communicator’s construction dtype.
- property nvlink_kernel: str | None¶
"cake"(the exported Cake programs offlashinfer.cake_ulysses_backend),"portable"(FlashInfer’s portable NVLink kernel) orNonewhen the NVLink backend is not armed (NCCL backend,world_size == 1, closed).- Type:
Which kernel the NVLink backend launches
- prepare_qkv(qkv_shape: Sequence[int], *, quantization: str = 'sage2', used_sequence: int | None = None) UlyssesQKVWorkspace¶
Collectively prepare Sage2 QKV communication and reusable storage.
Every rank must call this method in the same order, outside CUDA graph capture. Shape/configuration, layout/JIT availability, and allocation outcomes are agreed in three separate stages. A recoverable preparation failure releases only this call’s storage; it does not close the communicator or invalidate existing workspaces. Missing participants and CUDA/process-group failures are not recoverable through this protocol.
This method does not change
backend. Quantized QKV always uses the group’s CUDA NCCL backend, even when ordinary scatter/gather uses NVLink. The experimental PCIe backend is not supported by this prepared API. Supports homogeneous SM90 or SM120 groups of 2/4/8 ranks.- Parameters:
qkv_shape (Sequence[int]) – Common local
[B, L, H, D]shape, with positive dimensions,Dequal to 64 or 128, andHdivisible by the group size. Each Q/K/V and the combined byte payload must fit the int32 element-index limit. The complete packed QKV payload, including scales and alignment, must fitmax_bytes. Statistics are allocated separately.quantization (str, default = "sage2") – QKV encoding policy. Only
"sage2"is supported: grouped INT8 Q/K and channel-scaled FP8 E4M3 V, using the device’s Sage2 layout. The communicator dtype must be FP16 or BF16.used_sequence (int, optional) – Global live prefix length
Uin(0, world_size * L].Noneuses the whole sequence. Callers must zero input rows outside this prefix; this method does not initialize inputs.
- Returns:
Owner-bound, read-only geometry with reusable send/receive and statistics-gather buffers. Shape or live-length changes require another collective preparation call.
- Return type:
- scatter_heads(x: Tensor, *, out: Tensor | None = None, dtype: dtype | None = None, workspace: UlyssesWorkspace | None = None) Tensor¶
[B, S_local, H, D] -> [B, S_global, H_local, D].Scatter the global heads across ranks and gather the full sequence: afterwards this rank holds head slice
[rank * H_local, (rank+1) * H_local)of every token. Runs on the current CUDA stream. Returns the input unchanged whenworld_size == 1.- Parameters:
x (torch.Tensor) – Contiguous 4-D CUDA tensor with shape
[B, S_local, H, D].out (torch.Tensor, optional) – Preallocated contiguous output with shape
[B, S_local * world_size, H // world_size, D]. Supplying it removes the public output allocation. It must not aliasxwhenworld_size > 1. Multi-rank PCIe requires an output returned byallocate_output(); on the NVLink Cake route such an output takes the direct path.dtype (torch.dtype, optional) – Element type for this call, overriding the communicator dtype. pcie backend only.
max_bytesis denominated in bytes, so a narrower dtype here simply fits more elements in the same capacity.workspace (UlyssesWorkspace, optional) – Reusable NCCL pack/receive storage. Supplying it removes the two NCCL staging allocations. It is validated but unused by the NVLink backend, which owns IPC staging internally, and rejected by the multi-rank pcie backend, which has no staging buffers.
- Returns:
Tensor with shape
[B, S_global, H_local, D]on the same device and dtype asx.- Return type:
torch.Tensor
- scatter_qkv(q: Tensor, k: Tensor, v: Tensor, *, workspace: UlyssesQKVWorkspace, out: UlyssesQKV | None = None) UlyssesQKV¶
Quantize and exchange QKV for the prepared SageAttention2 layout.
Runs one FP32 statistics AllGather and one jointly packed uint8 AllToAll on
group, using NCCL independently ofbackend. Execution uses the current CUDA stream; all ranks must issue the same sequence of calls with valid inputs. There is no per-call metadata collective, implicit preparation, or precision fallback. Local statistics still allocate temporary tensors.- Parameters:
q (torch.Tensor) – FP16/BF16 CUDA tensors matching the prepared
[B,L,H,D]shape, dtype and device. D must be contiguous; the base pointer and each outer stride must preserve 16-byte row alignment. Projection views satisfying those constraints do not require copies. Input rows outside the prepared global live prefix must already be zero.k (torch.Tensor) – FP16/BF16 CUDA tensors matching the prepared
[B,L,H,D]shape, dtype and device. D must be contiguous; the base pointer and each outer stride must preserve 16-byte row alignment. Projection views satisfying those constraints do not require copies. Input rows outside the prepared global live prefix must already be zero.v (torch.Tensor) – FP16/BF16 CUDA tensors matching the prepared
[B,L,H,D]shape, dtype and device. D must be contiguous; the base pointer and each outer stride must preserve 16-byte row alignment. Projection views satisfying those constraints do not require copies. Input rows outside the prepared global live prefix must already be zero.workspace (UlyssesQKVWorkspace) – Workspace returned by this communicator’s
prepare_qkv. Must not be shared by concurrent operations or another communicator.out (UlyssesQKV, optional) – Reuse these six output tensors. Shapes, dtypes, device, layout, logical/live lengths and input dtype must match the preparation. Outputs must be contiguous and cannot overlap each other, the inputs, or workspace storage.
Noneallocates independent final storage that remains valid after later workspace reuse.
- Returns:
Pre-quantized Q/K/V and three FP32 scales, with consumer metadata. Returns
outitself when supplied. Attention uses the live prefix; the caller restores a full-length floating output before invokinggather_heads.- Return type:
- scatter_qkv_head_chunk(query: Tensor, key: Tensor, value: Tensor, *, head_offset: int, head_count: int, out: Tensor | None = None, workspace: UlyssesWorkspace | None = None) Tensor¶
Pack and scatter one Q/K/V head band with one collective.
query,keyandvaluehave shape[B, S_local, H, D]and may be independent positive-strided views. The output is a fused[B, S_global, head_count, 3 * D]payload; slicing its final dimension produces the three attention operands.The method is a transport primitive, not a pipeline scheduler. A caller that overlaps input and output communication must use separate communicators and workspaces. Runs on the current CUDA stream.
When the NCCL backend,
B == 1,out is Noneand a workspace is supplied, the returned tensor aliases the workspace receive buffer and remains valid only until that workspace is reused. Passoutfor an explicit lifetime.- Parameters:
query (torch.Tensor) – Positive-strided CUDA tensor with shape
[B, S_local, H, D].key (torch.Tensor) – Positive-strided CUDA tensor with the same shape, dtype, and device as
query.value (torch.Tensor) – Positive-strided CUDA tensor with the same shape, dtype, and device as
query.head_offset (int) – Start of the selected band within each destination rank’s
H // world_sizelocal heads.head_count (int) – Number of consecutive local heads in the selected band.
out (torch.Tensor, optional) – Preallocated contiguous result with shape
[B, S_local * world_size, head_count, 3 * D]. It must not aliasquery,key, orvalue.workspace (UlyssesWorkspace, optional) – Reusable staging storage large enough for the fused Q/K/V payload. A workspace cannot service concurrent collectives.
- Returns:
Fused Q/K/V payload with shape
[B, S_global, head_count, 3 * D].- Return type:
torch.Tensor
- class flashinfer.comm.UlyssesWorkspace(*, max_elems: int, dtype: dtype, device: device | str | int | None = None)¶
Bases:
objectReusable staging storage for NCCL Ulysses collectives.
A workspace owns two flat CUDA buffers: one for the packed all-to-all input and one for the receive result. Passing it to
UlyssesCommunicator.scatter_heads()orUlyssesCommunicator.gather_heads()removes the per-call NCCL staging allocations. The NVLink backend already owns IPC staging storage, so it validates but otherwise does not use this object.Workspaces are local allocations: they own no process group and create no streams. A workspace may be shared by serialized scatter/gather calls, but must not be used by concurrent collectives.
- Parameters:
max_elems (int) – Capacity of each staging buffer in elements.
dtype (torch.dtype) – Element dtype; float16, bfloat16, or float32.
device (torch.device or str or int, optional) – CUDA device.
Noneuses the current CUDA device.
- __init__(*, max_elems: int, dtype: dtype, device: device | str | int | None = None)¶
Initialize reusable NCCL send and receive staging buffers.
- Parameters:
max_elems (int) – Capacity of each staging buffer in elements.
dtype (torch.dtype) – Element dtype; float16, bfloat16, or float32.
device (torch.device or str or int, optional) – CUDA device.
Noneuses the current CUDA device.
- property recv_buffer: Tensor¶
Flat receive staging buffer (advanced/debugging use).
- property send_buffer: Tensor¶
Flat send staging buffer (advanced/debugging use).
Prepared Low-Precision QKV Exchange¶
The public Lowp interface is UlyssesCommunicator together with
UlyssesQKVWorkspace and UlyssesQKV. Statistics, quantization
and payload primitives are private implementation details. See the
design document for the numerical
contract and kernel-level validation.
prepare_qkv is a collective preparation method; all ranks call it in the
same order before attention. It allocates a dedicated mixed-dtype workspace
from the local QKV shape. Ordinary create_workspace remains rank-local.
scatter_qkv currently supports the SageAttention2 encoding only:
# The original floating Q/K/V total is a conservative wire-byte budget.
with UlyssesCommunicator(group, max_bytes=3 * q.nbytes, dtype=q.dtype,
device=q.device, backend="nccl") as comm:
workspace = comm.prepare_qkv(q.shape, used_sequence=used)
encoded = comm.scatter_qkv(q, k, v, workspace=workspace)
# A framework adapter calls Sage with encoded.q/k/v and the scales.
# It returns [B, P*L, H/P, D], in encoded.input_dtype, with a zero tail.
output = comm.gather_heads(sage_adapter(encoded))
# After the previous attention has consumed the buffers:
encoded = comm.scatter_qkv(q_next, k_next, v_next,
workspace=workspace, out=encoded)
The result contains INT8 Q/K, FP8 E4M3 V, three FP32 scales, the Sage layout, logical and used sequence lengths, and the original input dtype. It accepts FP16/BF16 inputs on homogeneous SM90 or SM120 groups, D=64/128, and P=2/4/8. The last input dimension is dense and each row must be 16-byte aligned; projection views need not be fully contiguous. The input padding after the used global prefix must already be zero.
Statistics use NCCL on the communicator’s group. The packed payload goes
through exchange_chunks; its current NCCL and NVLink backend implementations
both use NCCL without another layout conversion or payload copy.
workspace.transport reports that actual transport. Preparation checks NCCL
capability even for an NVLink communicator. The prepared Lowp API accepts
NCCL and NVLink communicators; it collectively rejects the experimental PCIe
backend before JIT or workspace allocation.
max_bytes must cover the full combined Q/K/V payload, including scales and
chunk alignment. Preparation computes and validates the exact requirement;
one floating Q tensor’s byte size is generally insufficient. The example’s
3 * q.nbytes covers both supported layouts. NCCL preparation allocates the
actual payload size, not the full budget. Statistics and final outputs are
additional allocations, so max_bytes is not a total memory budget.
out=None returns independent storage. out= requires matching tensor
geometry and metadata, including the exact used length and original dtype.
Output fields must not overlap one another, inputs or workspace scratch.
A workspace belongs to its creating communicator and cannot be used after
that communicator closes. Calls are serialized on the current CUDA stream.
Prepared QKV workspaces and ordinary staging workspaces are not interchangeable.
- class flashinfer.comm.UlyssesQKV(q: Tensor, k: Tensor, v: Tensor, q_scale: Tensor, k_scale: Tensor, v_scale: Tensor, layout: str, logical_sequence: int, used_sequence: int, input_dtype: dtype)¶
Pre-quantized SageAttention2 operands returned by
scatter_qkv.qandkare contiguous INT8[B, S, H/P, D]tensors.vis FP8 E4M3[B, D, H/P, S_pad]in the consumer’s permuted layout. Q/K scales are FP32[B, H/P, width]; V scales are FP32[B, H/P, D]. Scale widths useused_sequence, whereas Q/K/V storage coverslogical_sequence.layoutidentifies the Sage consumer:"sage2_sm90"or"sage2_sm89_sm120".The record is immutable, but its six tensors may be overwritten by an explicit
out=call. Attention and its softmax scale remain the caller’s responsibility;input_dtypeis the floating output dtype.- input_dtype: dtype¶
Alias for field number 9
- k: Tensor¶
Alias for field number 1
- k_scale: Tensor¶
Alias for field number 4
- layout: str¶
Alias for field number 6
- logical_sequence: int¶
Alias for field number 7
- q: Tensor¶
Alias for field number 0
- q_scale: Tensor¶
Alias for field number 3
- used_sequence: int¶
Alias for field number 8
- v: Tensor¶
Alias for field number 2
- v_scale: Tensor¶
Alias for field number 5
- class flashinfer.comm.UlyssesQKVWorkspace¶
Prepared Lowp geometry and reusable communication storage.
Created collectively by
UlyssesCommunicator.prepare_qkv(); direct construction is not supported. Metadata is read-only. The workspace is bound to its owner communicator and cannot be used after that owner is closed, on another communicator, or concurrently across streams.qkv_shape,world_size,dtype,device,used_sequence,quantizationandlayoutdescribe the prepared operation.transportis always"nccl", independently of the ordinary scatter/gather backend. Storage follows PyTorch tensor lifetimes and owns neither an extra process group nor a separate close protocol.- property logical_sequence: int¶
Full sequence length, including any zero-padded suffix.
Low-Precision Implementation Primitives¶
The following internal helpers support numerical tests, trace definitions,
and kernel benchmarks. They are not exported from flashinfer.comm;
applications should use the prepared UlyssesCommunicator QKV API above.
|
Describe availability of the ulysses_lowp kernels on the current device. |
|
Return the headerless V2-G destination-chunk layout in bytes. |
|
Compute local FP32 K sum and V absolute maximum over sequence. |
|
Per-touched-global-group |
|
Per-touched-global-group |
|
Extract the |
|
Overwrite this rank's boundary slots with the cross-rank max merge. |
|
Per-channel raw-K min/max of this rank's two K boundary slices. |
|
Overwrite this rank's K boundary slots with the derived cross-rank amax. |
|
Deterministically zero every scale slot and the alignment tail. |
|
Quantize Q on the global grid directly into the V2-G payload. |
|
Quantize K (global grid) and V (global per-channel) into the payload. |
|
Fused V2-G quantize-and-pack convenience entry point. |
|
Unpack source chunks into pre-quant compute inputs on the global grid. |
|
Everything the single boundary-statistics AllGather must carry. |
|
Turn the gathered statistics into final quantization inputs. |
|
Quantize and pack using the finalized global-grid Q/K amax and V scale. |
|
Debug/test-only check: every cross-boundary scale slot is bit-identical on all sources that carry it. |
|
Quantize canonical NHD V to canonical E4M3 bit patterns. |
Head-Chunk Layout and Transport Primitives¶
Head-chunk APIs are opt-in building blocks for framework-owned
communication/compute pipelines. They do not choose a chunk schedule, create
streams/events or process groups, or alter the ordinary scatter_heads /
gather_heads path.
pack_ulysses_qkv_head_chunk accepts independent positive-strided Q/K/V
views, selects the same destination-local head band from every Ulysses rank’s
head slice, and fuses Q/K/V into the last dimension. The communicator’s
scatter_qkv_head_chunk method combines that transform with one all-to-all.
After attention, gather_output_head_chunk performs the reverse all-to-all
and writes the compact band directly into the full output. A framework that
overlaps the two communication directions must use separate communicators and
workspaces:
schedule = ((0, 3), (3, 8), (11, 3)) # local_heads == 14
full_output = torch.empty_like(q)
for head_offset, head_count in schedule:
qkv = input_comm.scatter_qkv_head_chunk(
q, k, v,
head_offset=head_offset,
head_count=head_count,
out=qkv_chunk_buffers[head_count],
workspace=input_workspace,
)
q_chunk, k_chunk, v_chunk = qkv.split(head_dim, dim=-1)
o_chunk = attention(q_chunk, k_chunk, v_chunk)
output_comm.gather_output_head_chunk(
o_chunk,
local_heads=14,
head_offset=head_offset,
out=full_output,
workspace=output_workspace,
)
Head chunking is not automatically profitable. In particular, tensor
parallelism reduces the effective local head count to approximately
H / (TP * Ulysses). Frameworks should enable a schedule only after all
ranks agree that the attention backend supports every chunk head count and an
offline benchmark shows a positive result. Small local-head counts, GQA/MLA
with unequal Q/K/V heads, Ring Attention, and Attention2D should use the
ordinary path unless separately implemented and measured.
- flashinfer.comm.pack_ulysses_qkv_head_chunk(query: Tensor, key: Tensor, value: Tensor, *, world_size: int, head_offset: int, head_count: int, out: Tensor | None = None) Tensor¶
Pack one destination-local head band of Q/K/V.
The three inputs use
[B, S_local, H, D]and may be independent positive-strided views (including views produced by a fused projection).Hmust be divisible byworld_size. The selected local band is applied inside every destination rank’s head slice, and the result is a contiguous fused payload[B, S_local, world_size * head_count, 3 * D].The payload can be passed directly to
flashinfer.comm.UlyssesCommunicator.scatter_heads(); its result has shape[B, S_global, head_count, 3 * D]. Slicing the final dimension yields Q/K/V views for the attention backend.This operation runs on the caller’s current CUDA stream and allocates only when
outis omitted.- Parameters:
query (torch.Tensor) – Positive-strided CUDA tensor with shape
[B, S_local, H, D].key (torch.Tensor) – Positive-strided CUDA tensor with the same shape, dtype, and device as
query.value (torch.Tensor) – Positive-strided CUDA tensor with the same shape, dtype, and device as
query.world_size (int) – Number of Ulysses ranks.
Hmust be divisible byworld_size.head_offset (int) – Start of the selected band within every destination rank’s
H // world_sizelocal heads.head_count (int) – Number of consecutive local heads in the selected band.
out (torch.Tensor, optional) – Preallocated contiguous result with shape
[B, S_local, world_size * head_count, 3 * D]. It must not aliasquery,key, orvalue.
- Returns:
Contiguous fused Q/K/V payload with shape
[B, S_local, world_size * head_count, 3 * D].- Return type:
torch.Tensor
- flashinfer.comm.merge_ulysses_output_head_chunk(received: Tensor, *, world_size: int, local_heads: int, head_offset: int, out: Tensor) Tensor¶
Merge one compact Ulysses output head band into a full output.
receivedis the result of gathering a chunked attention output and has shape[B, S_local, world_size * head_count, D].outis the full[B, S_local, world_size * local_heads, D]destination. Only the band[rank * local_heads + head_offset, + head_count)for each source rank is written; other output elements remain unchanged.Calling this function once for every non-overlapping chunk in a schedule whose head counts sum to
local_headsreconstructs the ordinary whole-head Ulysses result. It runs on the caller’s current CUDA stream and performs no allocation.- Parameters:
received (torch.Tensor) – Contiguous gathered head band with shape
[B, S_local, world_size * head_count, D].world_size (int) – Number of Ulysses ranks represented in
receivedandout.local_heads (int) – Number of attention-output heads contributed by each rank to the full output.
head_offset (int) – Start of the selected band within each rank’s
local_heads.out (torch.Tensor) – Preallocated contiguous destination with shape
[B, S_local, world_size * local_heads, D]. It must not aliasreceived; elements outside the selected band remain unchanged.
- Returns:
outafter merging the selected head band.- Return type:
torch.Tensor
Fused QKV Projection + Norm + RoPE + Head Exchange¶
QkvNormRopeExchangeOverlap fuses the attention front end of a
sequence-parallel (Ulysses) layer into one prepared session: the BF16 QKV
projection, per-head RMSNorm of Q and K (one [D] scale each), interleaved
RoPE from a per-token [rows, D] cos/sin table, and the head exchange. The
Cake backend (backend="nvlink"; SM90, SM100, SM103 and SM107, single-node NVLink
groups of 2, 4 or 8 ranks (2 or 4 on SM107), head_dim=128, BF16) stores every destination’s
head band straight from the GEMM epilogue into that rank’s symmetric receive
slot and publishes per-chunk readiness words, so the exchange overlaps the
projection instead of following it; a one-CTA wait kernel on a consumer stream
acquires the words and the caller’s stream joins it. The outputs are views of
the current receive slot (layout="bshd": (q, k, v) each
[B, S_global, H_local, D]; layout="fused": one
[B, S_global, H_local, 3 * D] payload) and stay valid until the slot is
reused num_slots calls later; pass out= for an owned copy.
Per call the Cake backend launches, by default on every architecture, the
producer build with the folded cross-rank rendezvous (a rendezvous warp of
CTA 0 runs the NVLink barrier of the call’s phase and releases the stores
through a rank-local arrive flag: one extra warp per CTA on SM100 / SM103, the
idle TMA warp on SM90) on the caller’s stream and the wait kernel on the
consumer stream: two launches. fold_barrier=False (or the environment
variable FLASHINFER_QKV_EXCHANGE_FOLD_BARRIER=0) selects the three-launch
path instead: the barrier kernel, the plain producer and the wait kernel; both
paths are delivered on every architecture. Both paths store identical bits;
describe() reports the resolved path (fold_barrier) and its launch
count (launches_per_call).
The producer build is selected per architecture and world size
(cake_qkv_norm_rope_exchange_overlap.PRODUCER_VARIANT_DEFAULT_BY_ARCH_WORLD
of the JIT loader): on SM100 a group of 8 ranks runs the build with wide
operand loads, bulk row stores and the fused multiply-add epilogue
(bulk_ow256_fma), a group of 4 ranks the build with wide operand loads and
the fused multiply-add epilogue (ow256_fma) and a group of 2 ranks the
baseline build with the fused multiply-add epilogue (fma); on SM103 groups
of 8 and 4 ranks run ow256_fma and groups of 2 ranks fma; on SM107 a
group of 4 ranks runs the build with bulk row stores (bulk); the other
groups of 2 ranks run the baseline build; SM90 runs the build whose epilogue
stages each head in shared memory for three storer warps (sw3) on every
group size, except that under this default a group of 8 ranks on SM90 runs the
baseline build on rows with a hidden size of at least 6144 and at least 8192
rows per rank (cake_qkv_exchange_plan.PRODUCER_SHIPPED_ROWS_BY_ARCH_WORLD);
a build named by FLASHINFER_QKV_EXCHANGE_PRODUCER_VARIANT runs on every
row. The fma builds contract
the norm and RoPE multiply-adds into single-rounding fused multiply-adds, so
their stored bits can differ from those of the builds without it; both agree
within the kernel’s accuracy tolerance, and the builds without the fused
multiply-add epilogue store identical bits. Every build requires cos_sin
and norm-weight tensors that start at 16-byte-aligned addresses, the wide-load
builds at 32-byte-aligned ones (a misaligned view raises; for the 32-byte bases
the error names the opt-out). The
environment variable FLASHINFER_QKV_EXCHANGE_PRODUCER_VARIANT (auto /
shipped / a delivered build name) overrides the selection per session,
rank-uniformly; the baseline build stays delivered beside every variant. A
default build that is not delivered for the session’s tile width (the two-rank
fma build at a 128-column tile, for example hidden 768 with 6 heads over 2
ranks) runs the baseline build instead; a build named by the environment
variable that is not delivered for the tile width raises and names the
delivered ones. The host plan’s group scaling follows the same keying
(cake_qkv_exchange_plan.GROUP_M_SCALE_DEFAULT_BY_ARCH_WORLD): on SM100
groups of 4 and 2 ranks and on SM103 every group size double the token-tile
group of a grouped row whose hidden size is at least 6144 (the tile order
changes, every destination element is still written once); SM100 groups of 8
ranks, SM90 and SM107 keep the baseline plan. FLASHINFER_QKV_EXCHANGE_GROUP_M_SCALE
(auto / 1 / 2 / 4) overrides it per session, rank-uniformly.
Per-architecture row-count gates shape each call’s host plan: on SM100 and
SM103 a call with at most 4096 rows per rank publishes one readiness word per
destination, and a call with at least a threshold of rows per rank launches the
producer once per destination run of its tile table (the first launch carrying
the folded rendezvous); the threshold depends on the world size and the hidden
size (cake_qkv_exchange_plan.SHAPE_GATE_K_TABLE: 8192, 16384 or 32768 rows,
per_dest_k_table=0 restores the single threshold of 16384 and an explicit
per_dest_large_m replaces the table). On SM103 and SM100 a group of 8 ranks
runs a call whose destination band is not a multiple of 256 columns while every
Q / K / V group is (for example hidden 3072) with at least 2048 rows per rank
on the 256-column pair tile across band edges, each tile storing its two heads
to their own destinations, as one persistent launch
(cake_qkv_exchange_plan.XBAND_MIN_M_DEFAULT_BY_ARCH_WORLD;
xband_min_m=0 switches it off, and it stays off where its producer build
is not delivered, for example under the shipped producer build). On SM90 a
call with at most 4096 rows per rank publishes one
readiness word per destination, a call with at least 16384 rows per rank runs the
persistent producer grid over every SM (no reserved SMs), and the per-destination
launch gate is off; SM107 has no per-architecture gate. Three cells differ from these
per-architecture tables (cake_qkv_exchange_plan.SHAPE_GATE_DEFAULT_BY_ARCH_WORLD):
a group of 8 ranks on SM90 and groups of 2 and 4 ranks on SM107 publish one
readiness word per destination for every row count
(words_capacity_small_m = all). A fourth gate,
pair_off_large_m, is off on every architecture: switched on
(pair_off_large_m=N) it runs a 256-column call with at least N rows per
rank on the single-CTA producer instead of the 2-CTA pair tile (the tile order
is unchanged); it needs that producer delivered for the architecture – no
delivered Blackwell route ships it today, so the session refuses at
construction when a 256-column call can reach the gate (max_tokens >= N),
naming the gate – and is refused on SM90, which builds no pair tile. The
environment variable FLASHINFER_QKV_EXCHANGE_SHAPE_GATES (auto / off /
words_capacity_small_m=N|all,reserved_sms_large_m=N,per_dest_large_m=N,pair_off_large_m=N,
plus per_dest_k_table=0|1 and xband_min_m=N)
overrides them per session, rank-uniformly (words_capacity_small_m=4096 on
SM90 with 8 ranks and words_capacity_small_m=0 on SM107 restore the previous
plans); describe() reports the resolved
build (producer_variant), scale (group_m_scale) and gates
(shape_gates, shape_gate_k_table, xband_min_m) with their sources, and every prepared row’s verdict
(prepared_row_plans, including its reserved_sms). The first three gates
change the host plan only – neither the tile table nor the stored bits; the
pair-tile gate changes the producer program and the tile table (no lockstep
pads) of the rows it selects; no gate changes the stored bits. The largest
2-rank shape of the evaluation grid (S_global = 131072, hidden = 8192)
is not supported on an 80 GB H100: its operands and receive slots exceed the
device memory.
The session is prepared eagerly (symmetric slots, readiness pad, barrier
flags, JIT programs, the tile plan of max_tokens rows) and a prepared call
is CUDA-graph capturable: a captured call allocates nothing, runs no
collective, publishes a sentinel epoch and clears its readiness words with a
stream memset after the wait kernel, and replays interleave with eager calls.
A call with more rows than max_tokens, or with a row count first seen
inside a capture, raises instead of allocating. backend="auto" selects
the Cake path when resolve_ulysses_backend() verifies the NVLink mesh and
the device is a supported architecture, and otherwise the NCCL path (cuBLAS
projection, flashinfer.norm.rmsnorm(), RoPE, then
UlyssesCommunicator.scatter_qkv_head_chunk()), which is not graph
capturable (graph_capturable reports False).
|
Prepared session of the fused QKV projection + norm + RoPE + head exchange on one process group. |
- class flashinfer.comm.QkvNormRopeExchangeOverlap(group: ProcessGroup | None = None, *, hidden: int, num_heads: int, head_dim: int = 128, dtype: dtype = torch.bfloat16, layout: str = 'bshd', max_tokens: int, num_slots: int = 2, backend: str = 'auto', device: device | str | int | None = None, fold_barrier: bool | None = None)¶
Bases:
objectPrepared session of the fused QKV projection + norm + RoPE + head exchange on one process group.
Every rank constructs the session collectively (same arguments, same order) and calls it with its own
[B, S_local, hidden]activation shard; each rank receives head band[rank * H_local, (rank + 1) * H_local)of the whole sequence.- Parameters:
group (torch.distributed.ProcessGroup, optional) – NCCL process group of 2, 4 or 8 ranks on one node (
NoneusesWORLD).hidden (int) – Model width
num_heads * head_dim(a multiple of 64).num_heads (int) – Attention heads; must be divisible by the world size.
head_dim (int, default 128) – Head dimension (the fused kernels fix 128).
dtype (torch.dtype, default torch.bfloat16) – Operand dtype (BF16 only).
layout ({"bshd", "fused"}) –
"bshd":(q, k, v)each[B, S_global, H_local, head_dim];"fused": one[B, S_global, H_local, 3 * head_dim]payload.max_tokens (int) – Largest
B * S_localof any call; sizes the symmetric receive slots once (collective).num_slots (int, default 2) – Receive slots cycled by consecutive calls; a call’s outputs stay valid for
num_slots - 1further calls.backend ({"auto", "nvlink", "nccl"}) –
"nvlink"selects the Cake fused kernels (single-node NVLink mesh, SM90 / SM100 / SM103 / SM107, groups of 2, 4 or 8 ranks (2 or 4 on SM107), torch’s default CUDA symmetric-memory backend) and raises when they cannot be used;"nccl"selects the per-operator path (cuBLAS,rmsnorm, RoPE,UlyssesCommunicator.scatter_qkv_head_chunk);"auto"resolves throughresolve_ulysses_backend()and the device architecture.device (torch.device or str or int, optional) – CUDA device of this rank (
None: the current device).fold_barrier (bool, optional) – Launch path of the Cake backend.
True: the producer program with the folded cross-rank rendezvous – two launches per call (producer, wait kernel);False: the separate barrier kernel before the producer – three launches.None(default) reads the environment variableFLASHINFER_QKV_EXCHANGE_FOLD_BARRIER(0/1/auto) and otherwise takes the architecture default:Trueon SM90, SM100, SM103 and SM107 (the folded producers are delivered on every architecture). Both paths store identical bits. Must be uniform across the ranks of the group; not accepted withbackend="nccl".
Notes
The Cake backend selects its producer build per architecture and world size (
flashinfer.jit.cake_qkv_norm_rope_exchange_overlap.PRODUCER_VARIANT_DEFAULT_BY_ARCH_WORLD: SM100 groups of 8 ranks run thebulk_ow256_fmabuild – wide operand loads, bulk row stores and the fused multiply-add epilogue –, SM100 groups of 4 ranks and SM103 groups of 8 or 4 ranks theow256_fmabuild – wide operand loads and the fused multiply-add epilogue –, SM100 and SM103 groups of 2 ranks thefmabuild – the fused multiply-add epilogue –, SM107 groups of 4 ranks thebulkbuild – bulk row stores –, SM90 thesw3build – the epilogue stages each head in shared memory for three storer warps –, the other groups the baseline build). Under this default, SM90 groups of 8 ranks run the baseline build on rows withhidden >= 6144andbatch * s_local >= 8192(cake_qkv_exchange_plan.PRODUCER_SHIPPED_ROWS_BY_ARCH_WORLD); a build named by the environment variable below runs on every row. The builds without the fused multiply-add epilogue store identical bits; anfmabuild rounds the norm and RoPE multiply-adds once, so its stored bits can differ from theirs within the kernel’s accuracy tolerance. Every build requirescos_sinand norm-weight tensors that start at 16-byte-aligned addresses, the wide-load builds at 32-byte-aligned ones (a misaligned view raises; for the 32-byte bases the error names the opt-out). The environment variableFLASHINFER_QKV_EXCHANGE_PRODUCER_VARIANT(auto/shipped/ a delivered build name) overrides the selection for a session, rank-uniformly; the baseline build is delivered beside every variant. A default build that is not delivered for the session’s tile width (the two-rankfmabuild atblock_n=128, e.g. hidden 768 with 6 heads over 2 ranks) runs the baseline build instead; a build named by the environment variable that is not delivered for the tile width raises and names the delivered ones. The host plan’s group scaling follows the same keying (cake_qkv_exchange_plan.GROUP_M_SCALE_DEFAULT_BY_ARCH_WORLD:2on SM100 groups of 4 and 2 ranks and on SM103,1otherwise – SM100 groups of 8 ranks, SM90 and SM107; it applies to grouped rows withhidden >= 6144and changes the tile order only);FLASHINFER_QKV_EXCHANGE_GROUP_M_SCALE(auto/1/2/4) overrides it for a session, rank-uniformly.The Cake backend also applies per-architecture row-count gates to each call’s host plan (
cake_qkv_exchange_plan.SHAPE_GATE_DEFAULT_BY_ARCH; SM100 / SM103: a call withbatch * s_local <= 4096rows publishes one readiness word per destination, and the per-destination producer launch threshold followscake_qkv_exchange_plan.SHAPE_GATE_K_TABLEby world size and hidden size (8192, 16384 or 32768 rows);per_dest_k_table=0restores 16384 and an explicitper_dest_large_moverrides the table. SM90: a call with<= 4096rows publishes one readiness word per destination and a call with>= 16384rows runs the persistent producer grid over every SM –reserved_sms_large_m = 16384, no reserved SMs –, the per-destination launch gate off; SM107: no per-architecture gate). Three (architecture, world size) cells differ from these tables (cake_qkv_exchange_plan.SHAPE_GATE_DEFAULT_BY_ARCH_WORLD): a group of 8 ranks on SM90 and groups of 2 and 4 ranks on SM107 publish one readiness word per destination for every row count (words_capacity_small_m = all). A fourth gate,pair_off_large_m, is off on every architecture: switched on, a 256-column call with at leastNrows per rank runs the single-CTA producer instead of the 2-CTA pair tile (tile order unchanged; that producer must be delivered for the architecture – no delivered Blackwell route ships it today, so the session refuses at construction when a 256-column call can reach the gate,max_tokens >= N, naming the gate; SM90 builds no pair tile and refuses the gate). The environment variableFLASHINFER_QKV_EXCHANGE_SHAPE_GATES(auto/off/words_capacity_small_m=N|all,reserved_sms_large_m=N,per_dest_large_m=N,pair_off_large_m=N,per_dest_k_table=0|1,xband_min_m=N,0= never) overrides them for a session, rank-uniformly (words_capacity_small_m=4096on SM90 with 8 ranks andwords_capacity_small_m=0on SM107 restore the previous plans). On SM100 and SM103 the per-destination launch gate reads its threshold by hidden size (cake_qkv_exchange_plan.SHAPE_GATE_K_TABLE;per_dest_k_table=0in the same variable restores the one-band threshold, an explicitper_dest_large_mreplaces the table). SM103 and SM100 groups of 8 ranks run a call whose destination band is not a multiple of 256 columns while every Q / K / V group is (e.g. hidden 3072) with the 256-column pair tile across band edges from 2048 rows per rank on, as one persistent launch (cake_qkv_exchange_plan.XBAND_MIN_M_DEFAULT_BY_ARCH_WORLD;xband_min_m=Nin the same variable sets the threshold,xband_min_m=0switches it off). Neither changes the stored bits.describe()reports the resolved build (producer_variant), scale (group_m_scale), gates (shape_gates,shape_gate_k_table,xband_min_m), each with its value, source, environment variable and default, and every prepared row’s verdict (prepared_row_plans, includingreserved_smsandpart_grids). The first three gates change the host plan only – neither the tile table nor the stored bits; the pair-tile gate changes the producer program and the tile table (no lockstep pads) of the rows it selects; no gate changes the stored bits. The largest 2-rank shape of the evaluation grid (S_global = 131072,hidden = 8192) is not supported on an 80 GB H100: its operands and receive slots exceed the device memory.- __init__(group: ProcessGroup | None = None, *, hidden: int, num_heads: int, head_dim: int = 128, dtype: dtype = torch.bfloat16, layout: str = 'bshd', max_tokens: int, num_slots: int = 2, backend: str = 'auto', device: device | str | int | None = None, fold_barrier: bool | None = None) None¶
- bound_programs(batch: int, s_local: int) dict[str, str]¶
Cake backend: program names of every delivered stage of a
(batch, s_local)call (prepares the row eagerly);describe()["launch_sequence"]says which of them the launch path launches.
- close() None¶
Release the session’s resources (collective for the Cake backend).
- last_outputs()¶
Views of the most recent call’s outputs.
- poison_receive_slots() None¶
Fill this rank’s receive region with a NaN pattern (eager only), so the next call must rewrite it.
Topology Probing and Backend Selection¶
|
Outcome of backend selection. |
|
Per-rank topology probe result, exchanged across the group. |
|
Group-wide route plan for the explicit PCIe backend. |
Raised when a forced topology-specific backend cannot be supported. |
The communicator exposes .backend (the effective backend), .transport
("p2p", "hybrid" or "rdma", PCIe only), .decision /
.topology_decision and .fallback_reason.
- flashinfer.comm.resolve_ulysses_backend(backend: str = 'auto', group: ProcessGroup | None = None, device: device | str | int | None = None) UlyssesBackendDecision¶
Group-consistent backend selection. Must run before any IPC allocation or JIT compilation. It allocates no IPC workspace and compiles nothing; the
all_gather_objectmetadata collectives may themselves stage through CUDA buffers on NCCL process groups.Collective-safe outcome protocol: every rank participates in the same fixed prefix of
all_gather_objectcalls (at most three; a group-wide explicit NCCL request or an invalid/inconsistent request exits jointly after the first) no matter what fails locally — rank-local errors are encoded as serializable outcomes and re-raised (or turned into an NCCL fallback) jointly after the gather, so no rank can leave the collective sequence early and deadlock its peers. The only uncoordinated failure mode left is the process group itself failing.Sequence:
gather every rank’s requested backend; jointly reject invalid or inconsistent requests. A group-wide explicit
"nccl"request returns here, skipping the CUDA/NVML topology probe entirely (the gather itself may stage through CUDA on NCCL groups).gather every rank’s probe outcome (the probe never raises; even a buggy probe implementation is caught into the outcome).
every rank evaluates the same pure decision on the same gathered list, catches the result into an outcome, gathers, and cross-checks. Any disagreement conservatively selects NCCL — or raises for an explicitly forced
backend="nvlink"/backend="pcie".
- Parameters:
backend (str, default = "auto") – Requested backend policy:
"auto","nvlink","pcie", or"nccl".group (torch.distributed.ProcessGroup, optional) – Process group whose ranks must make a consistent backend decision. Defaults to
torch.distributed.group.WORLD.device (torch.device or str or int, optional) – CUDA device to use for probe and metadata collective guards.
- Returns:
Group-consistent backend choice and selection or fallback reason.
- Return type:
- flashinfer.comm.decide_ulysses_backend(requested: str, topologies: List[UlyssesRankTopology], supported_world_sizes: Sequence[int] = (2, 4, 6, 8), allow_experimental_auto: bool = False) UlyssesBackendDecision¶
Pure decision function: gathered per-rank probes -> (backend, reason).
Deterministic in its inputs, so every rank that gathers the same topology list computes the same decision. Conservative: anything unknown, inconsistent, or unverifiable selects NCCL. Raises
UlyssesBackendErroronly when"nvlink"or"pcie"is explicitly requested and that path cannot be used.- Parameters:
requested (str) – Requested backend policy:
"auto","nvlink","pcie", or"nccl".topologies (list[UlyssesRankTopology]) – Per-rank topology probe results gathered from the process group.
supported_world_sizes (Sequence[int], optional) – World sizes for which the fused NVLink kernel is instantiated.
allow_experimental_auto (bool, optional) – Whether
"auto"may select the experimental PCIe backend. Passed in rather than read from the environment here so this stays a pure function of its arguments: the caller resolvesFLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDSonce and gathers it across ranks, so the group cannot split on a per-rank setting.
- Returns:
Backend choice and selection or fallback reason.
- Return type:
- flashinfer.comm.probe_ulysses_rank_topology(device: device | int | str | None, rank: int, *, probe_pcie: bool = True) UlyssesRankTopology¶
Probe this rank’s GPU identity and its P2P/NVLink reachability to every other CUDA device visible to this process.
Never raises: the whole probe (including hostname and device resolution) runs inside an exception envelope, so any failure lands in
probe_errorand the (conservative) decision layer falls back to NCCL.- Parameters:
device (torch.device or str or int, optional) – CUDA device for this rank.
Noneuses the current CUDA device.rank (int) – Rank id to record in the returned topology object.
probe_pcie (bool) – Also walk sysfs for the NUMA node, mlx5 NIC and RoCE v2 GID this rank would use on the PCIe backend, filling
numa_node,nic_name,gid_indexandpcie_error/gid_error. Onlybackend="pcie"consumes these; passFalseto skip the walk.
- Returns:
Per-rank topology information, including any probe error instead of raising locally.
- Return type:
Raw Kernel Entry Points (advanced)¶
Prefer UlyssesCommunicator; these assume the caller has already
verified all-pairs NVLink P2P and owns the IPC workspace lifecycle.
|
Initialize the fused-transpose Ulysses NVLink-P2P all-to-all backend. |
Release a handle returned by |
|
|
Fused-transpose Ulysses all-to-all. |
MNNVL (Multi-Node NVLink)¶
MnnvlMemory and SymmDeviceMemory allocate the symmetric memory behind
the NVLink collectives. The symmetric memory design notes
describe how each lays out its memory and how the ranks choose between FABRIC
handles and POSIX file descriptors to share it.
Core Classes¶
|
|
|
Wrapper class for SymmDeviceMemory to facilitate PyTorch tensor creation. |
TensorRT-LLM MNNVL AllReduce¶
|
Deprecated pointer-based MNNVL all-reduce API. |
|
Perform an MNNVL all-reduce sum across tensor-parallel ranks. |
Performs MNNVL Allreduce + Residual + RMSNorm. |
|
Perform MNNVL AllReduce + Residual + RMSNorm + FP8/NVFP4/MXFP8 quantization. |
|
Performs MNNVL TwoShot Allreduce + RMSNorm. |
|
DCP All-to-All (Context-Parallel Attention Reduction)¶
|
Return the workspace size (in bytes) per rank for the given CP group size. |
Allocate an MNNVL-backed workspace of shape |
|
|
Initialize the workspace FIFO buffers (call once before the first alltoall). |
|
Perform the DCP all-to-all exchange. |
NCCL LSA DCP All-to-All + LSE Reduce¶
The fused path uses PyTorch NCCL symmetric memory and requires every context-
parallel rank to be in one load/store-accessible NVLink domain. It does not use
the MNNVL workspace accepted by decode_cp_a2a_alltoall.
Return the required NCCL symmetric workspace size in bytes. |
|
Create and rendezvous the fused op's NCCL symmetric workspace. |
|
|
Fuse an NCCL LSA DCP A2A exchange with the LSE-weighted reduce. |
TP MoE Finalize + RMSNorm¶
Tensor-parallel MoE finalize fused with the all-reduce and per-expert RMSNorm,
over an NVLink multicast workspace (hidden size 4096, top-k 8). Initialize the
process-wide workspace collectively with init_tp_moe_finalize_rmsnorm before
the first call.
|
Collectively initialize the process-wide TP multicast workspace. |
|
Return whether tensors satisfy the TP SM100/SM103/SM107 kernel contract and |
|
TP MoE finalize plus per-expert RMSNorm, for hidden size 4096 and top-k 8. |
MNNVL CuTe DSL Fused Collectives¶
Mailbox, stage state and compiled kernels for one TP group. |
|
Check only this rank. |
|
|
Per-expert RMSNorm finalize of local routes, all-reduced into |
Check only this rank. |
|
Gather column shards across TP, then the gated residual and next RMSNorm. |
|
Check only this rank. |
|
|
Gather column shards across TP, then RMSNorm and NVFP4-quantize the rows. |
Mixed Communication¶
|
Enumeration of mixed communication operation types. |
|
Enumeration of mixed communication execution modes. |
|
An implementation for the combinations of all-reduce + all-gather and reduce-scatter + all-reduce. |
|
Execute a mixed communication operation. |