flashinfer.quantization.mxfp8_grouped_quantize¶
- flashinfer.quantization.mxfp8_grouped_quantize(a: Tensor, mask: Tensor, backend: Literal['cutile', 'cake', 'cuda'] = 'cutile') Tuple[Tensor, Tensor]¶
Quantize grouped inputs to MXFP8 with UE8M0 block scales.
- Parameters:
a (torch.Tensor) – Input tensor of shape
[B, M, K]with dtypefloat16orbfloat16.mask (torch.Tensor) – Int32 CUDA tensor of shape
[B]with the number of valid rows of each group. It is read only on the device, so its values are not validated. The"cuda"backend uses the firstmin(mask[i], M)rows. The"cutile"and"cake"backends require0 <= mask[i] <= M; withmask[i] > Mthe cuTile kernel writes out of bounds.backend ({
"cutile","cake","cuda"}) –Implementation backend.
"cutile"(default) requires thecuda-tilepackage and"cake"a generated profile for the exact Blackwell architecture."cake"and"cuda"require contiguousaandmask;"cuda"also requiresato be 32-byte aligned. A failed launch is not retried on another backend.Outputs match across backends for finite inputs, including bfloat16 subnormals. A zero UE8M0 scale byte produces signed-zero payloads.
"cuda"quantizes NaN and infinities to 448, asmxfp8_quantize()does; blocks that hold them can differ.
- Returns:
(x_q, sf)wherex_qhas logical shape[M, padded_K, B]with dtypefloat8_e4m3fnandsfhas logical shape[32, 4, padded_M // 128, 4, padded_K // 128, B]with dtypeuint8.padded_KroundsKup to a multiple of 128 andpadded_MroundsMup to a multiple of 128. The physical layouts are grouped byBand then permuted to match FlashInfer masked grouped GEMM conventions.Rows past the valid ones (and their scale factors) are unspecified.
- Return type:
Tuple[torch.Tensor, torch.Tensor]