flashinfer.quantization.mxfp8_grouped_quantize

flashinfer.quantization.mxfp8_grouped_quantize(a: Tensor, mask: Tensor, backend: Literal['cutile', 'cake', 'cuda'] = 'cutile') → Tuple[Tensor, Tensor]

Quantize grouped inputs to MXFP8 with UE8M0 block scales.

Parameters:
  • a (torch.Tensor) – Input tensor of shape [B, M, K] with dtype float16 or bfloat16.

  • mask (torch.Tensor) – Int32 CUDA tensor of shape [B] with the number of valid rows of each group. It is read only on the device, so its values are not validated. The "cuda" backend uses the first min(mask[i], M) rows. The "cutile" and "cake" backends require 0 <= mask[i] <= M; with mask[i] > M the cuTile kernel writes out of bounds.

  • backend ({"cutile", "cake", "cuda"}) –

    Implementation backend. "cutile" (default) requires the cuda-tile package and "cake" a generated profile for the exact Blackwell architecture. "cake" and "cuda" require contiguous a and mask; "cuda" also requires a to be 32-byte aligned. A failed launch is not retried on another backend.

    Outputs match across backends for finite inputs, including bfloat16 subnormals. A zero UE8M0 scale byte produces signed-zero payloads. "cuda" quantizes NaN and infinities to 448, as mxfp8_quantize() does; blocks that hold them can differ.

Returns:

(x_q, sf) where x_q has logical shape [M, padded_K, B] with dtype float8_e4m3fn and sf has logical shape [32, 4, padded_M // 128, 4, padded_K // 128, B] with dtype uint8. padded_K rounds K up to a multiple of 128 and padded_M rounds M up to a multiple of 128. The physical layouts are grouped by B and then permuted to match FlashInfer masked grouped GEMM conventions.

Rows past the valid ones (and their scale factors) are unspecified.

Return type:

Tuple[torch.Tensor, torch.Tensor]