flashinfer.page.fp8_token_head_quantize_append_paged_kv_cache_with_slot_mapping¶
- flashinfer.page.fp8_token_head_quantize_append_paged_kv_cache_with_slot_mapping(append_key: Tensor, append_value: Tensor, slot_mapping: Tensor, paged_kv_cache: Tensor | Tuple[Tensor, Tensor], kv_cache_sf: Tuple[Tensor, Tensor], kv_layout: str = 'NHD', kv_data_type: str | dtype = 'e4m3') None¶
Quantize and write K/V rows into an FP8 per-(token, head) paged KV cache.
This is the write-side counterpart to the FA2 per-(token, head) FP8 KV scale read path (
use_token_head_sf=True). For each row the kernel computes a device-side scalemax(amax(head) / fp8_max, 1e-12), writes the normalized FP8 data into the paged slot, and writes the float32 scale into the caller’s scale view. The FP8 data and the scale share one inline slot: the slot’s last dimension ishead_dim + 16=[head_dim FP8][4-byte scale][12-byte pad]. Supportedhead_dimvalues are 64, 128, 256 and 512.slot_mapping[i]ispage_id * page_size + entry_idx; negative slots are padding and are ignored.append_key/append_valuemay hold extra padded rows; onlyslot_mapping.shape[0]rows are written.- Parameters:
append_key (torch.Tensor) – The key to quantize and write, shape
[num_rows, num_kv_heads, head_dim]with dtypetorch.float16ortorch.bfloat16.append_value (torch.Tensor) – The value to quantize and write, same shape and dtype as
append_key.slot_mapping (torch.Tensor) – Flat cache slot for each row to write, shape
[nnz]with dtypetorch.int32ortorch.int64. Negative entries are ignored.paged_kv_cache (Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]) – Caller-owned uint8 inline-slot K/V cache. For tuple input each tensor has shape
[max_num_pages, page_size, num_kv_heads, head_dim + 16]whenkv_layout="NHD"and[max_num_pages, num_kv_heads, page_size, head_dim + 16]whenkv_layout="HND".kv_cache_sf (Tuple[torch.Tensor, torch.Tensor]) –
[k_sf, v_sf]float32 scale views mirroring the KV layout minushead_dim(3-D paged:[max_num_pages, page_size, num_kv_heads]NHD /[max_num_pages, num_kv_heads, page_size]HND). These may be non-contiguous views sliced out ofpaged_kv_cache.kv_layout (str) – Layout of the paged KV cache, either
"NHD"or"HND".kv_data_type (Union[str, torch.dtype]) – FP8 element type of the written data,
"e4m3"(default) or"e5m2". Must match thekv_data_typeused by the read path.
- Returns:
This function updates
paged_kv_cacheandkv_cache_sfin place.- Return type:
None