flashinfer.gemm.mm_fp16_fp8¶
- flashinfer.gemm.mm_fp16_fp8(a: Tensor, b: Tensor, alpha: Tensor, *, bias: Tensor | None = None, out_dtype: dtype | None = None, out: Tensor | None = None, backend: Literal['auto', 'cute-dsl'] = 'auto', tactic: str | None = None, workspace: Tensor | None = None) Tensor¶
float16 GEMM whose
[N, K]operand is stored as FP8 e4m3 (W8A16).out = a @ (b.float() * alpha).T + biaswith the weight dequantized inside the kernel:mm_bf16_fp8()forfloat16activations. Same kernels, same arguments, same contract, withfloat16wherever that function saysbfloat16:ais(..., K)float16;biasandoutarefloat16;out_dtypeaccepts onlytorch.float16; the result is(..., N)infloat16. The e4m3 weight is converted tofloat16exactly, the products are accumulated in fp32, andalphaandbiasare applied before the single rounding intofloat16.tactictakes the same keys (MM_BF16_FP8_TACTICS_SM100/MM_BF16_FP8_TACTICS_SM90);mm_fp16_fp8_workspace_bytes()sizes theworkspace. Seemm_bf16_fp8()for the full description of every argument, the shape, alignment and aliasing rules, and the errors.