Welcome to FlashInfer’s documentation!

Blog | Discussion Forum | GitHub

FlashInfer is a library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA. FlashInfer focus on LLM serving and inference, and delivers state-of-the-art performance across diverse scenarios.

Get Started

• Installation
  • Python Package
  • C++ API

Tutorials

• Attention States and Recursive Attention
  • Applications
  • Related APIs
• KV-Cache Layout in FlashInfer
  • Layout: NHD/HND
  • Ragged Tensor
  • Mask Layout (2D Ragged Tensor)
  • Page Table Layout
  • Multi-level Cascade Inference Data Layout
  • FAQ

PyTorch API Reference

• flashinfer.decode
  • Single Request Decoding
  • Batch Decoding
• flashinfer.prefill
  • Single Request Prefill/Append Attention
  • Batch Prefill/Append Attention
• flashinfer.cascade
  • Merge Attention States
  • Cascade Attention
  • Cascade Attention Wrapper Classes
• flashinfer.sparse
  • BlockSparseAttentionWrapper
• flashinfer.page
  • Append new K/V tensors to Paged KV-Cache
• flashinfer.sampling
  • flashinfer.sampling.sampling_from_probs
  • flashinfer.sampling.top_p_sampling_from_probs
  • flashinfer.sampling.top_k_sampling_from_probs
  • flashinfer.sampling.min_p_sampling_from_probs
  • flashinfer.sampling.top_k_top_p_sampling_from_logits
  • flashinfer.sampling.top_k_top_p_sampling_from_probs
  • flashinfer.sampling.top_p_renorm_probs
  • flashinfer.sampling.top_k_renorm_probs
  • flashinfer.sampling.top_k_mask_logits
  • flashinfer.sampling.chain_speculative_sampling
• flashinfer.gemm
  • FP8 Batch GEMM
  • Grouped GEMM
• flashinfer.norm
  • flashinfer.norm.rmsnorm
  • flashinfer.norm.fused_add_rmsnorm
  • flashinfer.norm.gemma_rmsnorm
  • flashinfer.norm.gemma_fused_add_rmsnorm
• flashinfer.rope
  • flashinfer.rope.apply_rope_inplace
  • flashinfer.rope.apply_llama31_rope_inplace
  • flashinfer.rope.apply_rope
  • flashinfer.rope.apply_llama31_rope
  • flashinfer.rope.apply_rope_pos_ids
  • flashinfer.rope.apply_rope_pos_ids_inplace
  • flashinfer.rope.apply_llama31_rope_pos_ids
  • flashinfer.rope.apply_llama31_rope_pos_ids_inplace
  • flashinfer.rope.apply_rope_with_cos_sin_cache
  • flashinfer.rope.apply_rope_with_cos_sin_cache_inplace
• flashinfer.activation
  • Up/Gate output activation
• flashinfer.quantization
  • flashinfer.quantization.packbits
  • flashinfer.quantization.segment_packbits