flashinfer.gemm.mm_bf16_fp8¶
- flashinfer.gemm.mm_bf16_fp8(a: Tensor, b: Tensor, alpha: Tensor, *, bias: Tensor | None = None, out_dtype: dtype | None = None, out: Tensor | None = None, backend: Literal['auto', 'cute-dsl'] = 'auto', tactic: str | None = None, workspace: Tensor | None = None) Tensor¶
bfloat16 GEMM whose
[N, K]operand is stored as FP8 e4m3 (W8A16).out = a @ (b.float() * alpha).T + biaswith the weight dequantized inside the kernel; no 16-bit copy of the weight is allocated. “fp8” says how the weight is stored (e4m3 plus one per-tensor fp32 scale); the arithmetic is a 16-bit GEMM inbfloat16.mm_fp16_fp8()is the same kernel for float16 activations.Numerics. Each e4m3 weight value is converted to
bfloat16exactly (every finite e4m3 value is representable in both bf16 and fp16, denormals included), the unscaled products are accumulated in fp32, andalphaandbiasare applied to the fp32 accumulator before the single rounding to the output dtype. The result is therefore not bitwise equal to a GEMM against a weight pre-scaled and rounded to the activation dtype ((b.to(dtype) * alpha) @ ...): that carries one rounding more. The weight must be finite. Values are not checked, and a NaN e4m3 encoding yields an unspecified result (a finite value or NaN).- Parameters:
a (torch.Tensor) – Activation, shape
(..., K),bfloat16, contiguous, on a CUDA device (Hopper or Blackwell; every other tensor must be on the same device). Leading dimensions are flattened intoM; a leading dimension of 0 returns the empty output without launching.b (torch.Tensor) – Weight, shape
(N, K)(torch.nn.Linearlayout),float8_e4m3fn, contiguous, ona’s device.alpha (torch.Tensor) – One
float32value (shape()or(1,)) ona’s device: the per-tensor dequantization scale ofb.bias (Optional[torch.Tensor]) – Optional bias, shape
(N,),bfloat16, ona’s device, contiguous.out_dtype (Optional[torch.dtype]) – Output dtype. Only
bfloat16(a’s dtype, the default) is supported: the kernels round the fp32 accumulator once into the activation dtype. The argument exists for uniformity with the othermm_*functions.out (Optional[torch.Tensor]) – Optional output, shape
(..., N)(a’s leading dimensions),bfloat16, ona’s device, contiguous. Allocated when omitted.backend (str) –
"auto"(default) or"cute-dsl", the one implementation there is (CuTe-DSL kernels for Hopper and Blackwell).tactic (Optional[str]) – Selects a tile or a split policy instead of the launcher’s choice: a key of
MM_BF16_FP8_TACTICS_SM100on Blackwell or ofMM_BF16_FP8_TACTICS_SM90on Hopper (<weight rows>x<activation rows>[_streamk|_splitk]; the tables serve both activation dtypes).None= the launcher decides. This is the only way to override the choice; no environment variable does (the JIT / cache variables under Notes do not affect selection). A plain name runs every output tile whole on that tile shape. A_splitkname cuts the tiles into K-slices (fp32 partials in the workspace); it is rejected with a ValueError on aKit cannot slice (Hopper:Ka multiple of 64 whose 64-deep tile count divides into 2..8 equal slices of at least 8 tiles; Blackwell:Ka multiple of 128 whose 128-deep tile count has a divisor in 2..8), and runs the tile plain (no workspace) where the split cannot run (Hopper: a tail whose slices would not fit one wave of SMs; Blackwell:splits x Mat or above2**31). A_streamkname is a policy on the named tile, not a stream-K kernel: on Hopper it runs the whole waves of that tile plain and splits only the last partial wave over the idle SMs (plain when that tail is at least half a wave orKis short); on Blackwell it picks, by a wave-fill cost model, between the plain 2SM tiles and split-K on the 256x128 tile.mm_bf16_fp8_workspace_bytes()reports what the resolved policy needs.workspace (Optional[torch.Tensor]) – Optional scratch buffer: contiguous, 16-byte aligned, of any dtype, on
a’s device, at leastmm_bf16_fp8_workspace_bytes()bytes for this call’s shape, device andtactic(its byte size counts, not its dtype). Allocated per call when omitted and needed. A caller that wants a fixed memory budget sizes one buffer once and passes it to every call, under these rules: the call overwrites the buffer (nothing in it survives), it must not share memory with any other argument (the aliasing rule under Notes), and two calls may share one buffer only if they are ordered on one stream or synchronised, since a call still running on another stream would be reading what the next call overwrites.
Notes
Runs on Hopper (
sm_90a) and Blackwell (sm_100a/sm_103a);mm_bf16_fp8.is_compute_capability_supported(cc)answers for a device. The kernels are CuTe-DSL, compiled on first use of each kernel variant a call resolves to (so the first call of a new configuration is slow), and are not part of the AOT packages. Compiled kernels are cached on disk under the JIT workspace like every CuTe-DSL kernel, so a later process reloads instead of recompiling;FLASHINFER_DISABLE_JITandFLASHINFER_CUTE_DSL_DISABLE_CACHEapply.Shapes and layout:
Kmust be a multiple of 16 andNa multiple of 8, both non-zero, andM(the product ofa’s leading dimensions, not each of them),NandKeach at most2**31 - 512, withceil(M / 64) * ceil(N / 64)at most2**28 - 1(the kernels’ tile-count limit). Every tensor amonga/b/bias/out/workspacemust start at a 16-byte aligned address (a view into a larger buffer at an odd storage offset does not; contiguity alone is not enough). Nothing is copied on the caller’s behalf: a non-contiguousa,b,bias,outorworkspaceis rejected (call.contiguous()yourself if a copy is acceptable). Anything else is rejected with a ValueError before launch.Aliasing: the two buffers the call writes,
outandworkspace, must not share any bytes witha/b/alpha/biasor with each other; a call whose buffers overlap is rejected with a ValueError. The inputs may alias one another freely (they are only read).Empty calls: with
M == 0the dtypes, shapes, devices, contiguity and aliasing of every argument and the alignment ofb,biasandworkspaceare checked as for a non-empty call (the emptyaandouthave no address to align), then the empty output is returned without launching. No kernel variant is resolved and no workspace is sized or checked for such a call, so a split-Ktacticon an unsliceableKor an undersizedworkspaceis not reported there.- Returns:
out –
(..., N)inbfloat16.- Return type:
torch.Tensor
- Raises:
ValueError – Any argument outside the contract above (dtype, shape, device, contiguity, alignment, aliasing, an unknown
tacticorbackend, aworkspaceshorter than the query’s answer), before anything is launched.BackendSupportedError – A device the kernels do not support (anything but compute capability 9.0, 10.0 or 10.3).
RuntimeError – A launch failure reported by the CUDA runtime.