Welcome to FlashInfer’s documentation!

Blog | Discussion Forum | GitHub

FlashInfer is a library and kernel generator for Large Language Models that provides high-performance implementation of LLM GPU kernels such as FlashAttention, PageAttention and LoRA. FlashInfer focus on LLM serving and inference, and delivers state-of-the-art performance across diverse scenarios.

Get Started

• Installation
  • Python Package
• Command Line Interface
  • Quick Reference
  • Export Compile Commands
  • Module Management
  • Configuration and Status
  • Artifact Management
  • Cache Management
  • Replay Recorded Calls
• Logging
  • Quick Start
  • Logging Levels
  • Environment Variables
  • Miscellaneous Notes and Examples
  • Flight Recorder & Replay
  • Python-Based Replay Examples
• fi_trace — Operation Schema Extraction
  • Quick Start
  • Environment Variables
  • JSON File Format
  • Calling .fi_trace() Directly
  • Covered Operations
  • MoE Routing Types
  • Example: Collecting Traces from a Real Workload
  • Adding Trace Support to a New Kernel
  • Consistency Tests
• Trace Apply
  • Enabling
  • Routing
  • Solutions
  • Error policy
• Autotuning
  • What Is Autotuning?
  • Enabling Autotuning
  • Config Lookup Priority
  • Config Caching
  • API Reference
  • Multi-Thread / Multi-Process Considerations

Tutorials

• Attention States and Recursive Attention
  • Applications
  • Related APIs
• KV-Cache Layout in FlashInfer
  • Layout: NHD/HND
  • Ragged Tensor
  • Mask Layout (2D Ragged Tensor)
  • Page Table Layout
  • Multi-level Cascade Inference Data Layout
  • FAQ
• FlashInfer on JAX with TVM FFI
  • Execution requirements

PyTorch API Reference

• FlashInfer Attention Kernels
  • flashinfer.decode
  • flashinfer.prefill
  • flashinfer.mla
• flashinfer.gemm
  • BF16 GEMM
  • FP4 GEMM
  • MXFP8 GEMM
  • FP8 GEMM
  • Mixed Precision GEMM (fp8 x fp4)
  • Router GEMM (DeepSeek-V3 / Mistral / GLM)
  • Blackwell SM100 GEMM
  • Grouped GEMM (CuTe-DSL, Blackwell)
  • Grouped GEMM (Ampere/Hopper)
• flashinfer.grouped_mm
  • BF16 / FP16
  • FP8
  • MXFP8
  • FP4 (NVFP4 / MXFP4)
• flashinfer.fused_moe
  • Types and Enums
  • Utility Functions
  • Multi-LoRA MoE (BGMV)
  • CUTLASS Fused MoE
  • TensorRT-LLM Fused MoE
  • CuteDSL Fused MoE
• flashinfer.cascade
  • Merge Attention States
  • Cascade Attention
  • Cascade Attention Wrapper Classes
• flashinfer.comm
  • CUDA IPC Utilities
  • DLPack Utilities
  • Mapping Utilities
  • TensorRT-LLM AllReduce
  • Unified AllReduce Fusion API
  • vLLM AllReduce
  • MNNVL (Multi-Node NVLink)
  • TensorRT-LLM MNNVL AllReduce
  • MNNVL A2A (Throughput Backend)
  • DCP All-to-All (Context-Parallel Attention Reduction)
  • Mixed Communication
• flashinfer.sparse
  • BlockSparseAttentionWrapper
  • VariableBlockSparseAttentionWrapper
• flashinfer.pod
  • PODWithPagedKVCacheWrapper
  • BatchPODWithPagedKVCacheWrapper
• flashinfer.cudnn
  • flashinfer.cudnn.cudnn_batch_decode_with_kv_cache
  • flashinfer.cudnn.cudnn_batch_prefill_with_kv_cache
• flashinfer.cute_dsl
  • Availability
  • RMSNorm + FP4 Quantization
  • Attention Wrappers
• flashinfer.page
  • Append new K/V tensors to Paged KV-Cache
• flashinfer.sampling
  • flashinfer.sampling.sampling_from_probs
  • flashinfer.sampling.sampling_from_logits
  • flashinfer.sampling.softmax
  • flashinfer.sampling.top_p_sampling_from_probs
  • flashinfer.sampling.top_k_sampling_from_probs
  • flashinfer.sampling.min_p_sampling_from_probs
  • flashinfer.sampling.top_k_top_p_sampling_from_logits
  • flashinfer.sampling.top_k_top_p_sampling_from_probs
  • flashinfer.sampling.top_p_renorm_probs
  • flashinfer.sampling.top_k_renorm_probs
  • flashinfer.sampling.top_k_mask_logits
  • flashinfer.sampling.chain_speculative_sampling
• flashinfer.topk
  • Top-K Selection
  • Utility Functions
• flashinfer.logits_processor
  • Pipeline Construction
  • Pipeline
  • Processors
  • Types
  • Customization Features
• flashinfer.norm
  • flashinfer.norm.rmsnorm
  • flashinfer.norm.rmsnorm_quant
  • flashinfer.norm.fused_add_rmsnorm
  • flashinfer.norm.fused_add_rmsnorm_quant
  • flashinfer.norm.gemma_rmsnorm
  • flashinfer.norm.gemma_fused_add_rmsnorm
  • flashinfer.norm.layernorm
  • flashinfer.norm.fused_rmsnorm_silu
  • flashinfer.norm.fused_qk_rmsnorm_rope
  • flashinfer.norm.fused_dit_residual_layernorm_scale_shift
  • flashinfer.norm.fused_dit_gate_residual_layernorm_scale_shift
  • flashinfer.norm.fused_dit_gate_residual_layernorm_gamma_beta
• flashinfer.rope
  • flashinfer.rope.apply_rope_inplace
  • flashinfer.rope.apply_llama31_rope_inplace
  • flashinfer.rope.apply_rope
  • flashinfer.rope.apply_llama31_rope
  • flashinfer.rope.apply_rope_pos_ids
  • flashinfer.rope.apply_rope_pos_ids_inplace
  • flashinfer.rope.apply_llama31_rope_pos_ids
  • flashinfer.rope.apply_llama31_rope_pos_ids_inplace
  • flashinfer.rope.apply_rope_with_cos_sin_cache
  • flashinfer.rope.apply_rope_with_cos_sin_cache_inplace
  • flashinfer.rope.rope_quantize_fp8
  • flashinfer.rope.rope_quantize_fp8_append_paged_kv_cache
  • flashinfer.rope.mla_rope_quantize_fp8
• flashinfer.activation
  • Up/Gate output activation
• flashinfer.gdn_decode
  • flashinfer.gdn_decode.gated_delta_rule_decode
  • flashinfer.gdn_decode.gated_delta_rule_decode_pretranspose
  • flashinfer.gdn_decode.gated_delta_rule_mtp
• flashinfer.gdn_prefill
  • flashinfer.gdn_prefill.chunk_gated_delta_rule
• flashinfer.mamba
  • flashinfer.mamba.selective_state_update
  • flashinfer.mamba.checkpointing_ssu
• flashinfer.quantization
  • Types and Enums
  • Packbits Utilities
  • FP4 Quantization
  • FP8 Quantization
  • CuTe-DSL Quantization Kernels (experimental)
• flashinfer.green_ctx
  • Green context creation
• flashinfer.fp4_quantization
  • See Also
• flashinfer.testing
  • Test Environment Setup
  • Performance Analysis
  • GPU Benchmarking