flashinfer.page.fp8_token_head_quantize_append_paged_kv_cache_with_slot_mapping

flashinfer.page.fp8_token_head_quantize_append_paged_kv_cache_with_slot_mapping(append_key: Tensor, append_value: Tensor, slot_mapping: Tensor, paged_kv_cache: Tensor | Tuple[Tensor, Tensor], kv_cache_sf: Tuple[Tensor, Tensor], kv_layout: str = 'NHD', kv_data_type: str | dtype = 'e4m3') → None

Quantize and write K/V rows into an FP8 per-(token, head) paged KV cache.

This is the write-side counterpart to the FA2 per-(token, head) FP8 KV scale read path (use_token_head_sf=True). For each row the kernel computes a device-side scale max(amax(head) / fp8_max, 1e-12), writes the normalized FP8 data into the paged slot, and writes the float32 scale into the caller’s scale view. The FP8 data and the scale share one inline slot: the slot’s last dimension is head_dim + 16 = [head_dim FP8][4-byte scale][12-byte pad]. Supported head_dim values are 64, 128, 256 and 512.

slot_mapping[i] is page_id * page_size + entry_idx; negative slots are padding and are ignored. append_key/append_value may hold extra padded rows; only slot_mapping.shape[0] rows are written.

Parameters:
  • append_key (torch.Tensor) – The key to quantize and write, shape [num_rows, num_kv_heads, head_dim] with dtype torch.float16 or torch.bfloat16.

  • append_value (torch.Tensor) – The value to quantize and write, same shape and dtype as append_key.

  • slot_mapping (torch.Tensor) – Flat cache slot for each row to write, shape [nnz] with dtype torch.int32 or torch.int64. Negative entries are ignored.

  • paged_kv_cache (Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]) – Caller-owned uint8 inline-slot K/V cache. For tuple input each tensor has shape [max_num_pages, page_size, num_kv_heads, head_dim + 16] when kv_layout="NHD" and [max_num_pages, num_kv_heads, page_size, head_dim + 16] when kv_layout="HND".

  • kv_cache_sf (Tuple[torch.Tensor, torch.Tensor]) – [k_sf, v_sf] float32 scale views mirroring the KV layout minus head_dim (3-D paged: [max_num_pages, page_size, num_kv_heads] NHD / [max_num_pages, num_kv_heads, page_size] HND). These may be non-contiguous views sliced out of paged_kv_cache.

  • kv_layout (str) – Layout of the paged KV cache, either "NHD" or "HND".

  • kv_data_type (Union[str, torch.dtype]) – FP8 element type of the written data, "e4m3" (default) or "e5m2". Must match the kv_data_type used by the read path.

Returns:

This function updates paged_kv_cache and kv_cache_sf in place.

Return type:

None