flashinfer.quantization.nvfp4_kv_dequantize_paged

flashinfer.quantization.nvfp4_kv_dequantize_paged(paged_kv_cache: Tensor | Tuple[Tensor, Tensor], kv_cache_sf: Tensor | Tuple[Tensor, Tensor], block_tables: Tensor, seq_lens: Tensor, k_scale: Tensor, v_scale: Tensor, output_k: Tensor, output_v: Tensor, kv_layout: str = 'NHD') → None

Dequantize a paged NVFP4 KV cache into caller-owned contiguous outputs.

Requires SM80+. This helper gathers pages through block_tables and writes dequantized K/V tensors in [batch, max_seq_len, num_heads, head_dim] layout. Tokens at positions >= seq_lens[batch] are left unchanged.

Parameters:

  • paged_kv_cache (Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]) – The packed NVFP4 paged KV cache. Accepts the same tuple or stacked cache format as paged attention APIs. For tuple input, each tensor has shape [num_pages, page_size, num_kv_heads, head_dim // 2] when kv_layout="NHD" and [num_pages, num_kv_heads, page_size, head_dim // 2] when kv_layout="HND".

  • kv_cache_sf (Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]) – Per-block FP8 E4M3 scales with the same tuple or stacked cache format as paged_kv_cache, replacing head_dim // 2 with head_dim // 16.

  • block_tables (torch.Tensor) – Physical page table of shape [batch, max_pages_per_request] with dtype int32 or int64.

  • seq_lens (torch.Tensor) – Sequence lengths of shape [batch] with dtype int32.

  • k_scale (torch.Tensor) – Global dequantization scale tensors of dtype float32 on the same CUDA device as the cache.

  • v_scale (torch.Tensor) – Global dequantization scale tensors of dtype float32 on the same CUDA device as the cache.

  • output_k (torch.Tensor) – Caller-owned output tensors in [batch, max_seq_len, num_heads, head_dim] layout. Each must be contiguous and have dtype torch.float16 or torch.bfloat16.

  • output_v (torch.Tensor) – Caller-owned output tensors in [batch, max_seq_len, num_heads, head_dim] layout. Each must be contiguous and have dtype torch.float16 or torch.bfloat16.

  • kv_layout (str) – Layout of the paged input cache, either "NHD" or "HND".

Returns:

This function writes dequantized K/V values into output_k and output_v in place.

Return type:

None