flashinfer.prefill.fmha_v2_prefill_deepseek

flashinfer.prefill.fmha_v2_prefill_deepseek(query: Tensor, key: Tensor, value: Tensor, out: Tensor, num_heads: int, head_dim: int, seq_len: int, scale_softmax: float, scale_bmm1: float | None = None, scale_bmm2: float | None = None, return_lse: bool = False, lse: Tensor | None = None, scale_bmm1_d: Tensor | None = None, scale_bmm2_d: Tensor | None = None) → Tensor | Tuple[Tensor, Tensor]

Run causal SM12x FMHA v2 for MLA context or 128-dimension GQA.

Supports the DeepSeek Q/K=192, V=128 layout and FP8 E4M3 Q/K/V=128 with BF16 output, including GLM-4.7’s 96 query heads and 8 K/V heads. GLM-4.7-Flash MLA context uses Q/K/V=256 with 20 heads and supports FP8 E4M3 input with BF16 output. The 128- and 256-dimension paths accept MHA, GQA, and MQA with the same requirements as fmha_v2_prefill_sm120().

Parameters:
  • query (torch.Tensor) – query tensor with shape [batch_size, seq_len, num_heads, head_dim]

  • key (torch.Tensor) – key tensor with shape [batch_size, seq_len, num_kv_heads, head_dim]

  • value (torch.Tensor) – value tensor with shape [batch_size, seq_len, num_kv_heads, head_dim_v], where head_dim_v is 128 for Q/K=128 or 192, and 256 for Q/K=256.

  • out (torch.Tensor) – output tensor with shape [batch_size, seq_len, num_heads, head_dim_v]

  • return_lse (bool) – whether to return the log-sum-exp of attention output

  • num_heads (int) – number of heads

  • head_dim (int) – head dimension

  • seq_len (int) – sequence length

  • scale_softmax (float) – scale for softmax

  • scale_bmm1 (Optional[float]) – scale for bmm1

  • scale_bmm2 (Optional[float]) – scale for bmm2

  • lse (Optional[torch.Tensor]) – log-sum-exp of attention output

  • scale_bmm1_d (Optional[torch.Tensor]) – Optional persistent one-element FP32 CUDA model weight containing the FP8 QK scale. The host scale_bmm1 value is used when omitted.

  • scale_bmm2_d (Optional[torch.Tensor]) – Optional persistent one-element FP32 CUDA model weight containing the FP8 V dequantization scale. The host scale_bmm2 value is used when this is omitted.

Returns:

out – output torch.Tensor or Tuple[torch.Tensor, torch.Tensor]. If return_lse is True, the output will be a tuple of two tensors, the first is the output tensor, the second is the lse tensor. If return_lse is False, the output will be a single tensor.

Return type:

Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]