Welcome to FlashInfer’s documentation!

Blog | Discussion Forum | GitHub

FlashInfer is a library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA. FlashInfer focus on LLM serving and inference, and delivers state-of-the-art performance across diverse scenarios.

Get Started

• Installation
  • Python Package

Tutorials

• Attention States and Recursive Attention
  • Applications
  • Related APIs
• KV-Cache Layout in FlashInfer
  • Layout: NHD/HND
  • Ragged Tensor
  • Mask Layout (2D Ragged Tensor)
  • Page Table Layout
  • Multi-level Cascade Inference Data Layout
  • FAQ

PyTorch API Reference

• FlashInfer Attention Kernels
  • flashinfer.decode
  • flashinfer.prefill
  • flashinfer.mla
• flashinfer.gemm
  • FP4 GEMM
  • FP8 GEMM
  • Mixed Precision GEMM (fp8 x fp4)
  • Grouped GEMM (Ampere/Hopper)
• flashinfer.fused_moe
  • Types and Enums
  • Utility Functions
  • CUTLASS Fused MoE
  • TensorRT-LLM Fused MoE
• flashinfer.cascade
  • Merge Attention States
  • Cascade Attention
  • Cascade Attention Wrapper Classes
• flashinfer.comm
  • CUDA IPC Utilities
  • DLPack Utilities
  • Mapping Utilities
  • TensorRT-LLM AllReduce
  • vLLM AllReduce
  • MNNVL (Multi-Node NVLink)
  • TensorRT-LLM MNNVL AllReduce
• flashinfer.sparse
  • BlockSparseAttentionWrapper
  • VariableBlockSparseAttentionWrapper
• flashinfer.page
  • Append new K/V tensors to Paged KV-Cache
• flashinfer.sampling
  • flashinfer.sampling.sampling_from_probs
  • flashinfer.sampling.top_p_sampling_from_probs
  • flashinfer.sampling.top_k_sampling_from_probs
  • flashinfer.sampling.min_p_sampling_from_probs
  • flashinfer.sampling.top_k_top_p_sampling_from_logits
  • flashinfer.sampling.top_k_top_p_sampling_from_probs
  • flashinfer.sampling.top_p_renorm_probs
  • flashinfer.sampling.top_k_renorm_probs
  • flashinfer.sampling.top_k_mask_logits
  • flashinfer.sampling.chain_speculative_sampling
• flashinfer.logits_processor
  • Pipeline Construction
  • Pipeline
  • Processors
  • Types
  • Customization Features
• flashinfer.norm
  • flashinfer.norm.rmsnorm
  • flashinfer.norm.fused_add_rmsnorm
  • flashinfer.norm.gemma_rmsnorm
  • flashinfer.norm.gemma_fused_add_rmsnorm
  • flashinfer.norm.layernorm
• flashinfer.rope
  • flashinfer.rope.apply_rope_inplace
  • flashinfer.rope.apply_llama31_rope_inplace
  • flashinfer.rope.apply_rope
  • flashinfer.rope.apply_llama31_rope
  • flashinfer.rope.apply_rope_pos_ids
  • flashinfer.rope.apply_rope_pos_ids_inplace
  • flashinfer.rope.apply_llama31_rope_pos_ids
  • flashinfer.rope.apply_llama31_rope_pos_ids_inplace
  • flashinfer.rope.apply_rope_with_cos_sin_cache
  • flashinfer.rope.apply_rope_with_cos_sin_cache_inplace
• flashinfer.activation
  • Up/Gate output activation
• flashinfer.quantization
  • flashinfer.quantization.packbits
  • flashinfer.quantization.segment_packbits
• flashinfer.green_ctx
  • Green context creation
• flashinfer.fp4_quantization
  • Core Quantization Functions
  • Matrix Shuffling Utilities
  • Types and Enums
• flashinfer.testing
  • Test Environment Setup
  • Performance Analysis
  • GPU Benchmarking