flashinfer.gemm.mm_bf16_fp8

flashinfer.gemm.mm_bf16_fp8(a: Tensor, b: Tensor, alpha: Tensor, *, bias: Tensor | None = None, out_dtype: dtype | None = None, out: Tensor | None = None, backend: Literal['auto', 'cute-dsl'] = 'auto', tactic: str | None = None, workspace: Tensor | None = None) → Tensor

bfloat16 GEMM whose [N, K] operand is stored as FP8 e4m3 (W8A16).

out = a @ (b.float() * alpha).T + bias with the weight dequantized inside the kernel; no 16-bit copy of the weight is allocated. “fp8” says how the weight is stored (e4m3 plus one per-tensor fp32 scale); the arithmetic is a 16-bit GEMM in bfloat16. mm_fp16_fp8() is the same kernel for float16 activations.

Numerics. Each e4m3 weight value is converted to bfloat16 exactly (every finite e4m3 value is representable in both bf16 and fp16, denormals included), the unscaled products are accumulated in fp32, and alpha and bias are applied to the fp32 accumulator before the single rounding to the output dtype. The result is therefore not bitwise equal to a GEMM against a weight pre-scaled and rounded to the activation dtype ((b.to(dtype) * alpha) @ ...): that carries one rounding more. The weight must be finite. Values are not checked, and a NaN e4m3 encoding yields an unspecified result (a finite value or NaN).

Parameters:
  • a (torch.Tensor) – Activation, shape (..., K), bfloat16, contiguous, on a CUDA device (Hopper or Blackwell; every other tensor must be on the same device). Leading dimensions are flattened into M; a leading dimension of 0 returns the empty output without launching.

  • b (torch.Tensor) – Weight, shape (N, K) (torch.nn.Linear layout), float8_e4m3fn, contiguous, on a’s device.

  • alpha (torch.Tensor) – One float32 value (shape () or (1,)) on a’s device: the per-tensor dequantization scale of b.

  • bias (Optional[torch.Tensor]) – Optional bias, shape (N,), bfloat16, on a’s device, contiguous.

  • out_dtype (Optional[torch.dtype]) – Output dtype. Only bfloat16 (a’s dtype, the default) is supported: the kernels round the fp32 accumulator once into the activation dtype. The argument exists for uniformity with the other mm_* functions.

  • out (Optional[torch.Tensor]) – Optional output, shape (..., N) (a’s leading dimensions), bfloat16, on a’s device, contiguous. Allocated when omitted.

  • backend (str) – "auto" (default) or "cute-dsl", the one implementation there is (CuTe-DSL kernels for Hopper and Blackwell).

  • tactic (Optional[str]) – Selects a tile or a split policy instead of the launcher’s choice: a key of MM_BF16_FP8_TACTICS_SM100 on Blackwell or of MM_BF16_FP8_TACTICS_SM90 on Hopper (<weight rows>x<activation rows>[_streamk|_splitk]; the tables serve both activation dtypes). None = the launcher decides. This is the only way to override the choice; no environment variable does (the JIT / cache variables under Notes do not affect selection). A plain name runs every output tile whole on that tile shape. A _splitk name cuts the tiles into K-slices (fp32 partials in the workspace); it is rejected with a ValueError on a K it cannot slice (Hopper: K a multiple of 64 whose 64-deep tile count divides into 2..8 equal slices of at least 8 tiles; Blackwell: K a multiple of 128 whose 128-deep tile count has a divisor in 2..8), and runs the tile plain (no workspace) where the split cannot run (Hopper: a tail whose slices would not fit one wave of SMs; Blackwell: splits x M at or above 2**31). A _streamk name is a policy on the named tile, not a stream-K kernel: on Hopper it runs the whole waves of that tile plain and splits only the last partial wave over the idle SMs (plain when that tail is at least half a wave or K is short); on Blackwell it picks, by a wave-fill cost model, between the plain 2SM tiles and split-K on the 256x128 tile. mm_bf16_fp8_workspace_bytes() reports what the resolved policy needs.

  • workspace (Optional[torch.Tensor]) – Optional scratch buffer: contiguous, 16-byte aligned, of any dtype, on a’s device, at least mm_bf16_fp8_workspace_bytes() bytes for this call’s shape, device and tactic (its byte size counts, not its dtype). Allocated per call when omitted and needed. A caller that wants a fixed memory budget sizes one buffer once and passes it to every call, under these rules: the call overwrites the buffer (nothing in it survives), it must not share memory with any other argument (the aliasing rule under Notes), and two calls may share one buffer only if they are ordered on one stream or synchronised, since a call still running on another stream would be reading what the next call overwrites.

Notes

Runs on Hopper (sm_90a) and Blackwell (sm_100a / sm_103a); mm_bf16_fp8.is_compute_capability_supported(cc) answers for a device. The kernels are CuTe-DSL, compiled on first use of each kernel variant a call resolves to (so the first call of a new configuration is slow), and are not part of the AOT packages. Compiled kernels are cached on disk under the JIT workspace like every CuTe-DSL kernel, so a later process reloads instead of recompiling; FLASHINFER_DISABLE_JIT and FLASHINFER_CUTE_DSL_DISABLE_CACHE apply.

Shapes and layout: K must be a multiple of 16 and N a multiple of 8, both non-zero, and M (the product of a’s leading dimensions, not each of them), N and K each at most 2**31 - 512, with ceil(M / 64) * ceil(N / 64) at most 2**28 - 1 (the kernels’ tile-count limit). Every tensor among a / b / bias / out / workspace must start at a 16-byte aligned address (a view into a larger buffer at an odd storage offset does not; contiguity alone is not enough). Nothing is copied on the caller’s behalf: a non-contiguous a, b, bias, out or workspace is rejected (call .contiguous() yourself if a copy is acceptable). Anything else is rejected with a ValueError before launch.

Aliasing: the two buffers the call writes, out and workspace, must not share any bytes with a / b / alpha / bias or with each other; a call whose buffers overlap is rejected with a ValueError. The inputs may alias one another freely (they are only read).

Empty calls: with M == 0 the dtypes, shapes, devices, contiguity and aliasing of every argument and the alignment of b, bias and workspace are checked as for a non-empty call (the empty a and out have no address to align), then the empty output is returned without launching. No kernel variant is resolved and no workspace is sized or checked for such a call, so a split-K tactic on an unsliceable K or an undersized workspace is not reported there.

Returns:

out – (..., N) in bfloat16.

Return type:

torch.Tensor

Raises:
  • ValueError – Any argument outside the contract above (dtype, shape, device, contiguity, alignment, aliasing, an unknown tactic or backend, a workspace shorter than the query’s answer), before anything is launched.

  • BackendSupportedError – A device the kernels do not support (anything but compute capability 9.0, 10.0 or 10.3).

  • RuntimeError – A launch failure reported by the CUDA runtime.