Trace Apply¶
Trace Apply substitutes selected FlashInfer API calls with custom solutions at runtime, with no changes to the calling code (SGLang, vLLM, or other engines). It is the consumer side of the FlashInfer Trace: you provide a mapping from definition name to the solution to run for it, and Trace Apply dispatches that solution whenever the corresponding API is called with the matching shape. For any single definition there is exactly one solution — the one you registered.
Enabling¶
Register solutions explicitly with enable_apply(). The argument is a
mapping {definition_name: solution}, where a solution is either a Python
callable or a first-class Solution:
import torch
import flashinfer
import flashinfer.trace_apply as fi_trace_apply
def my_rmsnorm(hidden_states, weight, eps=1e-6):
x = hidden_states.float()
y = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + eps)
return (y * weight.float()).to(hidden_states.dtype)
fi_trace_apply.enable_apply({"rmsnorm_h4096": my_rmsnorm}) # returns the # of wrapped APIs
...
flashinfer.rmsnorm(x, w) # routed to my_rmsnorm when hidden_size == 4096
fi_trace_apply.stats() # per-API dispatch counts (hit / fallback / error)
fi_trace_apply.disable_apply() # restore the original FlashInfer APIs
enable_apply is idempotent — calling it again replaces the previous mapping.
With no argument (enable_apply()) it uses the environment configuration
below; if nothing is configured it is a no-op.
There is also an import-time hook controlled by environment variables:
Variable |
Meaning |
|---|---|
|
Set to |
|
Directory the deployment-configured solutions are loaded from. |
If the configuration is missing or invalid, Trace Apply stays disabled and FlashInfer continues to work normally, with a warning describing the problem.
FLASHINFER_TRACE_APPLY_PATH must point at a curated solutions folder — its
solutions/ subtree is scanned recursively for one solution per definition (a
duplicate definition is an error). It is not the raw extraction bundle, which
also contains baseline solutions and several backends per definition.
Routing¶
A call is routed by definition name. On each call the wrapper extracts the
call’s const axes and recomputes the definition name from the live
TraceTemplate — the same name_prefix + const-axis convention the trace
collector uses (e.g. rmsnorm at hidden_size=4096 → "rmsnorm_h4096").
If that name is in the registered mapping, the call dispatches to its solution;
otherwise it falls back to the original FlashInfer kernel.
Const axes are the compile-time shape a definition is specialized for (
hidden_size,head_dim, …) and form the name. Variable axes (batch size, sequence length, …) are not part of the name, so a single solution serves all of their values.The decision is cached per name, so steady-state dispatch is a dictionary lookup. During CUDA-graph capture only already-resolved shapes are applied, so warm up eagerly before capturing.
Solutions¶
A solution value is either:
a Python callable — invoked directly with the definition’s inputs by keyword (value-returning); or
a first-class
Solution— loaded by language family: the Python family (python,triton,cutedsl,cutile,tilelang) is imported and called by keyword; the C++/CUDA family (cpp,cuda,cutlass) is built viaflashinfer.jitand called positionally.
Trace Apply reconciles a solution’s outputs with the calling API’s convention:
value-returning outputs, caller-provided out= / lse= buffers, in-place
writes (e.g. fused_add_rmsnorm writing back into its input/residual buffers),
and data-dependent arity (e.g. return_lse). If a caller passes out= (e.g.
rmsnorm(x, w, out=buf)), the substituted solution’s result is written into
buf and buf is returned, exactly like the original kernel.
The out= / lse= bindings are auto-derived from the live API signature
(a uniform FlashInfer convention), so they are not recorded in the trace. Only
the non-derivable in-place bindings (which input a result is written back into)
are declared in the trace via the output param.
Error policy¶
Trace Apply is strict. A matched solution that fails to build or run raises, so a broken solution is reported immediately rather than masked. A call with no matching registered name falls back to the original FlashInfer API.
Setup: untimed weight preparation¶
A solution may consume weights in a layout of its own. Trace Apply never
converts a layout per call; instead the solution declares a setup step that
runs once per weight set, outside the call path and outside CUDA-graph capture,
and whose result reaches every later call by keyword. The declaration lives on the Solution spec and applies to the
curated path and to sealed packages alike:
Field |
Meaning |
|---|---|
|
|
|
|
|
The Definition inputs whose live tensors setup receives and whose identity keys the retained state: distinct identifiers, only with a setup. Empty means one state per process. |
|
The keyword the entry point takes the state under (default
|
The four fields join Solution.hash() and to_dict() only when a setup is
declared, so every existing solution, package and build-cache key is unchanged.
The ABI:
def setup(context): # once per weight set, eager, untimed
weights = context["setup_inputs"]["gemm1_weights"]
return {"relayout": my_layout(weights)} # or {"state": ..., "input_buffers": {}}
def run(hidden_states, gemm1_weights, ..., state, output):
... # Definition inputs, the state, DPS outputs
def teardown(state): # only on release_packages() / release()
...
context is a plain dict with these keys:
axes, device ("cuda:0"), rank and world_size (from
torch.distributed when initialized, else 0 and 1), inputs
({name: {"shape", "stride", "dtype", "device"}}: every Definition input in
declared order when a sealed bundle/definition.json is known, a scalar
input with shape None, else the call’s tensors),
input_values (the non-tensor inputs), setup_inputs (the live tensors)
and outputs. axes and outputs are filled from the same
definition.json, else empty; the curated loader passes no Definition.
A {"state", "input_buffers"} result envelope is accepted with empty
input_buffers and refused otherwise: serving has no harness copy phase.
A setup-bearing solution loads as a SetupBoundCallable
through import_entry, load_packages and the curated loader alike. Its
published signature (inspect.signature) is the entry point’s without the
state parameter, so an ABI check binds the Definition inputs exactly as before.
Per call it keys the setup_inputs tensors by identity — address, shape,
stride, dtype and device — and runs setup on a miss. A Definition name is a
shape class, not a layer: every MoE layer of a model shares one name and holds
its own weights, so one state per key is one state per weight set, and the view
of one storage an engine passes call after call keys the same. What setup
retains is a second resident copy, allocated at the first eager call of each
weight set and reported, never sized by Trace Apply. Identity cannot see bytes
change behind a stable address, so an engine mode that rewrites weights in
place is the consumer’s to refuse.
Eager before capture. A weight set first seen inside CUDA-graph capture is
refused with SetupStateError (a RuntimeError, never an ImportError)
and counted in refused_in_capture: setup allocates and synchronizes, and a
half-prepared call must never be captured. The first capture the callable
observes freezes its key set, so a weight set that appears afterwards is refused
too and a setup input that is really an activation cannot grow state without
bound (MAX_WEIGHT_SETS bounds it under eager execution). Run every weight
set eagerly before capturing, as an engine’s profiling and warm-up forwards do.
The check sees the calling thread’s capture (CUDA offers no process-wide
query), so a first use on one thread while another thread captures is what
that warm-up rule rules out, not something the callable can detect.
A state does not survive fork: a call from another process is refused. A
setup that calls its own Definition with a new weight set is refused too,
rather than left waiting on the lock it holds.
release_packages() first disarms the routes it releases: every sealed
Solution whose Definition no captured binding serves leaves Trace Apply’s
mapping (by identity, so a curated Solution enabled for that Definition since
stays, and the mapping is emptied in place, so a wrapper kept before its first
matching call routes nowhere; one that had already resolved keeps its
callable, which a released binding refuses), then loses its preloaded entry, so no route can fall
through to the on-disk build path at any point; curated Solutions enabled
beside them stay enabled. It then tears down every state those bindings retain
(teardown when declared; every binding is released before a failed
teardown is raised) and forgets them; a released binding refuses every later
call. Callers quiesce their calls first. setup_stats() returns {definition: {"runs", "weight_sets",
"seconds", "retained_bytes", "refused_in_capture"}} for the same bindings. A
binding that a CUDA-graph capture has observed is never released, by
release_packages() or by its own release() (a SetupStateError):
captured graphs replay against the state addresses, so the state is kept for
the life of the process and stays in setup_stats(). A setup-bearing
solution loaded through the curated path is bound the same way but registered
nowhere: disable_apply() drops it without teardown, and neither call sees
it. Nothing runs at process exit.
setup_state_param is a field rather than a fixed state because Trace
Apply calls Python-family solutions by keyword, so the wrapper must know the
parameter’s name; an existing kernel keeps whatever name it already uses, and
the Solution records it. setup_inputs sits on the Solution rather than on the
Definition because the Definition at this seam is a trace template with no
per-solution permission document. FlashInfer’s own weight-preparation helpers
(flashinfer.fused_moe.prepare) may be called from a setup: a solution that
needs the layout a FlashInfer kernel expects produces it there, once.
A complete setup-bearing package is preserved under
tests/cpu/trace_apply/data/kernel_package/48812d76b7d3073a961dc996d6968e09998bae895ff83f3a175fe7f98cb26280/:
bundle/kernel.py (run), bundle/prepare.py (setup and
teardown), bundle/solution.json with the four fields,
bundle/definition.json and a manifest.json in the sealed_files
spelling. The test suite verifies it, binds it and serves
flashinfer.norm.rmsnorm through it on the CPU.
Sealed kernel packages¶
The curated FLASHINFER_TRACE_APPLY_PATH folder is a development path: its
solutions are loaded as plain JSON with no verification. The production path
is a sealed kernel package — a directory whose name is the sha256 of its own
manifest.json, so the digest that names a package pins every byte of it
before anything is imported:
<root>/<sha256(manifest.json)>/
manifest.json the directory name is the sha256 of these bytes
<sidecar files> verified bytes, opaque to FlashInfer (e.g. a
qualification receipt)
bundle/
solution.json a FlashInfer Solution
... exactly the regular files the manifest lists
The digest chain, all of which must hold before any code runs:
the package directory name equals the sha256 of
manifest.json;filesmaps each bundle-relative file name to its sha256, with canonical path rules only (no absolute paths, no.., no NUL bytes);bundle_sha256equals the sha256 of the sorted"name\0digest\n"listing, so the listing itself cannot be edited to match a tampered file;the bundle holds exactly the files the manifest lists — a missing file, an extra file, a symbolic link, or a special file anywhere in the package refuses verification;
sidecar files outside the bundle are authenticated by
sealed_files(a mapping of package-relative names to sha256 digests). An earlier package format’squalification_sha256key is accepted as an alias forsealed_files = {"qualification.json": ...}, so existing packages verify unchanged. FlashInfer never interprets sidecar content: a consumer with domain metadata (a qualification receipt, provenance, licenses) reads the verified path throughSealedPackage.sealed_files;the
solution.jsonmust be a FlashInfer Solution in a Python-family language (python,triton,cutedsl,cutile,tilelang) — a compiled family would need nvcc at serving start, and a SOLBench/Kernel-Factoryspec.languagesdocument is refused rather than silently translated;every declared symbol (the entry point, and the setup and teardown of a setup declaration, Setup: untimed weight preparation) is validated at verification, with the standard library, before any sealed code runs: spelled
<file>.py::<symbol>(the formBuildSpectakes), a sealed Python file on a canonical path, not shadowed by a package, and listed in the Solution’ssources; a sealedbundle/definition.jsonmust name the solution’s definition, and sidecars are plain file names beside the manifest;teardown_entry_pointrequiressetup_entry_point,destination_passing_styleis explicit,setup_inputsare distinct identifiers and, whenbundle/definition.jsonis sealed, every setup input is a tensor input of it andsetup_state_paramis none of its inputs or outputs.
Loading¶
Three environment variables drive loading at import flashinfer time:
Variable |
Meaning |
|---|---|
|
Comma-separated sha256 digests of sealed packages to load and enable when FlashInfer is imported. |
|
Root directory holding the digest-named package
directories (default
|
|
Kill switch for the import-time loading above:
refuse it with a warning. A program calling
|
Loading is best-effort and never breaks import flashinfer: a refused
package warns (SealedPackageError is a RuntimeError, never an
ImportError, so no engine can mistake the refusal for a missing optional
dependency) and the surviving packages install. Packages may coexist with the
curated path, with a warning that the curated solutions are unverified.
Programmatic use¶
from flashinfer.trace_apply import load_packages, verify_package, import_entry
result = load_packages("/opt/flashinfer/kernel_packages", ["<digest>", ...])
result.packages # SealedPackage objects (sealed_files, solution, ...)
result.callables # {(definition, solution name): imported entry point}
result.errors # {digest: reason} for refusals when strict=False
result.applied # enable_apply() return value
A verified package’s entry point can also be imported on its own:
import_entry(verify_package(root, digest)) returns the sealed callable (for
a setup-bearing package, the entry point bound to its setup and teardown, both
imported from the same sealed namespace; see Setup: untimed weight preparation).
Imported callables are registered as preloaded (register_preloaded), so
the trace loaders never materialize sealed sources into the on-disk solution
cache and every deferred import re-hashes the bytes it compiles. Importing a
verified bundle executes its code: verification is content authentication, not
a sandbox.
Signatures¶
Kernels are not signed anywhere in FlashInfer today, and this feature does
not add signing: the digest chain authenticates content (what a digest names
is what runs), and the trust anchor is the party that names the digest. The
manifest reserves signature and schema_version fields so a future
signature mechanism (e.g. keyless sigstore over the manifest) can be added
without a format break; an unknown schema_version refuses.