Welcome to FlashInfer’s documentation!

Blog | Discussion Forum | GitHub

FlashInfer is a library for Language Languages Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA. FlashInfer focus on LLM serving and inference, and delivers state-of-the-art performance across diverse scenarios.

Get Started

• Installation
  • Python Package
  • C++ API

Tutorials

• Attention States and Recursive Attention
  • Applications
  • Related APIs
• KV-Cache Layout in FlashInfer
  • Layout: NHD/HND
  • Ragged Tensor
  • Mask Layout (2D Ragged Tensor)
  • Page Table

PyTorch API Reference

• flashinfer.decode
  • Single Request Decoding
  • Batch Decoding
• flashinfer.prefill
  • Single Request Prefill/Append Attention
  • Batch Prefill/Append Attention
• flashinfer.cascade
  • Merge Attention States
  • Cascade Attention
  • Cascade Attention Wrapper Classes
• flashinfer.page
  • Append new K/V tensors to Paged KV-Cache
• flashinfer.sampling
  • flashinfer.sampling.sampling_from_probs
  • flashinfer.sampling.top_p_sampling_from_probs
  • flashinfer.sampling.top_k_sampling_from_probs
  • flashinfer.sampling.top_p_renorm_prob
  • flashinfer.sampling.top_k_renorm_prob
  • flashinfer.sampling.chain_speculative_sampling
• flashinfer.group_gemm
  • SegmentGEMMWrapper
• flashinfer.norm
  • flashinfer.norm.rmsnorm
• flashinfer.quantization
  • flashinfer.quantization.packbits
  • flashinfer.quantization.segment_packbits