flashinfer.prefill.fmha_v2_prefill_deepseek¶
- flashinfer.prefill.fmha_v2_prefill_deepseek(query: Tensor, key: Tensor, value: Tensor, out: Tensor, num_heads: int, head_dim: int, seq_len: int, scale_softmax: float, scale_bmm1: float | None = None, scale_bmm2: float | None = None, return_lse: bool = False, lse: Tensor | None = None, scale_bmm1_d: Tensor | None = None, scale_bmm2_d: Tensor | None = None) Tensor | Tuple[Tensor, Tensor]¶
Run causal SM12x FMHA v2 for MLA context or 128-dimension GQA.
Supports the DeepSeek Q/K=192, V=128 layout and FP8 E4M3 Q/K/V=128 with BF16 output, including GLM-4.7’s 96 query heads and 8 K/V heads. GLM-4.7-Flash MLA context uses Q/K/V=256 with 20 heads and supports FP8 E4M3 input with BF16 output. The 128- and 256-dimension paths accept MHA, GQA, and MQA with the same requirements as
fmha_v2_prefill_sm120().- Parameters:
query (torch.Tensor) – query tensor with shape [batch_size, seq_len, num_heads, head_dim]
key (torch.Tensor) – key tensor with shape [batch_size, seq_len, num_kv_heads, head_dim]
value (torch.Tensor) – value tensor with shape [batch_size, seq_len, num_kv_heads, head_dim_v], where head_dim_v is 128 for Q/K=128 or 192, and 256 for Q/K=256.
out (torch.Tensor) – output tensor with shape [batch_size, seq_len, num_heads, head_dim_v]
return_lse (bool) – whether to return the log-sum-exp of attention output
num_heads (int) – number of heads
head_dim (int) – head dimension
seq_len (int) – sequence length
scale_softmax (float) – scale for softmax
scale_bmm1 (Optional[float]) – scale for bmm1
scale_bmm2 (Optional[float]) – scale for bmm2
lse (Optional[torch.Tensor]) – log-sum-exp of attention output
scale_bmm1_d (Optional[torch.Tensor]) – Optional persistent one-element FP32 CUDA model weight containing the FP8 QK scale. The host
scale_bmm1value is used when omitted.scale_bmm2_d (Optional[torch.Tensor]) – Optional persistent one-element FP32 CUDA model weight containing the FP8 V dequantization scale. The host
scale_bmm2value is used when this is omitted.
- Returns:
out – output torch.Tensor or Tuple[torch.Tensor, torch.Tensor]. If return_lse is True, the output will be a tuple of two tensors, the first is the output tensor, the second is the lse tensor. If return_lse is False, the output will be a single tensor.
- Return type:
Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]