flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss¶
- flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss(X: Tensor, W: Tensor, labels: Tensor, *, objective: str = 'ce', loss_div: float | None = None, infer_logp: Tensor | None = None, loss_weights: Tensor | None = None, chunk_size: int = 4096, return_logp: bool = False, grad_weight_dtype: dtype = torch.bfloat16, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False)¶
Differentiable chunked LM-head projection + loss with a memory-bounded backward, SM100 / SM103 / SM107 only.
Computes the projection
z = X @ W^Tover token chunks ofchunk_sizerows, the per-row log-probability of the selected tokenlogp_t = z[t, y_t] - logsumexp_v z[t, v]and one of two objectives, without ever materializing a vocabulary-sized tensor of more thanchunk_sizerows. The forward already produces the FP32 gradient accumulators of the trainable inputs (three GEMMs per chunk, no logits recomputation); the backward applies the incoming scalar gradient and performs the single cast to the output dtypes.- Parameters:
X (torch.Tensor) – BF16
[T, H]hidden states, row-major with any leading stride (Ha multiple of 256; a stride whose byte pitch is not a multiple of 16 is copied to a contiguous tensor first).W (torch.Tensor) – BF16
[V, H]output weight, contiguous (Va multiple of 256).labels (torch.Tensor) – int64
[T];-100marks an ignored row (zero loss, zero gradient,logp = 0).objective (str) –
"ce":loss = -sum(logp[valid]) / loss_div."policy":loss = -sum(loss_weights * min(exp(logp - infer_logp), 2))over valid rows; the per-row logit-gradient scale is-loss_weights_t * ratio_twhenratio_t <= 2and0above the clipping boundary.loss_div (Optional[float]) – Positive scalar divisor of the cross-entropy objective (caller-supplied, never replaced by a local token mean). Required for
"ce".infer_logp (Optional[torch.Tensor]) – FP32
[T]inference log-probabilities of the policy objective.loss_weights (Optional[torch.Tensor]) – FP32
[T]signed weights of the policy objective; they already include masking and normalization (ignored rows carry weight 0).chunk_size (int) – Token chunk
C(default 4096). Every logits / dlogits / probability buffer spans at mostCtokens; a batch smaller thanCis one chunk, and a tailT % Cis neither dropped nor padded. ChangingCmay change reduction rounding, never the loss definition.return_logp (bool) – Also return the detached FP32
logp [T](a diagnostic output).grad_weight_dtype (torch.dtype) – dtype of the weight gradient:
torch.bfloat16(default) ortorch.float32.dWis accumulated in FP32 across chunks and cast once at the output boundary. Through this autograd entry the value must equalW.dtype(the autograd engine casts every gradient to its leaf’s dtype); an FP32dWfor a BF16Wis available from the explicitcake_backend.forward_loss/backward_losspair.backend (str) – Only
"cake".compact_rows (bool, optional) – Chunk over the valid rows only (rows whose label is not
-100): the valid-row index is formed once per call, each chunk’s rows ofXare gathered into one reusable[chunk_size, H]buffer andlogp/dXare scattered back with exact zeros on the ignored rows – the same computation over fewer rows (per-rowlogpis bitwise that of the uncompacted path; theloss/dWreductions run over different chunk boundaries and differ by FP32 rounding).None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_COMPACT_ROWS(on unless set to0; the backend’s own default isTrue). Compaction costs one device synchronization per call, the valid-row count (the log-probability backward reuses the forward’s index).fuse_dw_cast (bool, optional) – Run the last chunk’s weight-gradient GEMM in the backward, where the upstream scalar gradient is known, with the scale and the output cast fused into its epilogue: the forward accumulates the chunks before it in FP32 (no accumulator at all for a one-chunk call), the backward writes
dW = cast(g * (dW_acc + dz_c^T @ X_c))straight from the GEMM and the separate pass over the[V, H]accumulator disappears. The same FP32 operations in the same order:dWis bitwise the unfused result. The last chunk’s BF16dlogitsrows and a view of its rows ofXare saved for the backward (an in-place write toXbetween the forward and the backward raises PyTorch’s saved-tensor version error).None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_FUSE_DW_CAST(on unless set to0; the backend’s own default isTrue).dx_finalize (bool, optional) – Add the K-slice slabs of a sliced
dXGEMM with one fixed-order kernel and, withcompact_rows, write the[T, H]dXin one pass (bf16(g * dX_acc)on the valid rows, exact zeros elsewhere) instead of casting, zero-filling and scattering the compact rows; the same FP32 operations in the same order, sodXis bitwise the same.None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_DX_FINALIZE(on unless set to0; the backend’s own default isTrue).dw_stream (bool or str, optional) – Launch each chunk’s weight-gradient accumulate GEMM on a per-device side stream, forked after the chunk’s row gradients and joined before the chunk buffer is reused and before the call returns, so it overlaps the tail of the chunk’s
dXGEMM:"auto"for calls of three or more chunks,Truefor every multi-chunk call,Falsenever; the same kernels in the same order per kernel, so every output is bitwise the same.None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_DW_STREAM(0/false/off/no->False,1/true/on/yes->True, anything else ->"auto"; the backend’s own default is"auto").hidden_count (bool, optional) – Form a compacted call’s valid-row index on the device and queue chunk 0’s row gather and logits GEMM (its device-count form, bounded by the count read from device memory) before the count reaches the host through a pinned cell and a CUDA event, instead of counting and indexing on the host before the first launch; the same kernels per row, so every output is bitwise the same.
None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_HIDDEN_COUNT(on unless set to0; the backend’s own default isTrue).gemm_tuning (dict, optional) – Explicit per-GEMM knobs,
{"logits": {...}, "dx": {...}, "dw": {...}}(e.g.{"dx": {"tile_n": 256, "k_slices": 2}}; a knob given asNonepins the default form), winning over the per-chunk rules for the knobs they name; a form the compiled program does not register fails closed.None(default) readsFLASHINFER_CAKE_LM_HEAD_LOSS_GEMM_TUNINGas a JSON object (unset or blank: no explicit knob, the backend’s own default; invalid JSON is aValueError).memory_mode (str, optional) –
None(the default): the automatic form. A call that forms a BF16dWon CUDA (grad mode on,Wrequiring a gradient) runs the fastest form while its predicted step peak, with that form’s own buffer lifetimes, stays withinBless the 2 % margin below, else the"budget"form the planner picks when one fits, else the fastest form (cake_backend_opt_in.auto_memory_plan). A call whose valid rows fit one chunk keeps the fastest form: with the fused weight-gradient cast (fuse_dw_cast, on by default) it holds no FP32[V, H]accumulator (without it, it does), and its step peak can exceedB(see the package README). It plans on the valid rows of the forward’s count or ofnum_valid_rows; an uncompacted call withoutnum_valid_rowskeeps the fastest form, as does every call without such adW."fast": always the fastest form, whose FP32[V, H]weight-gradient accumulator lives next to the chunk buffers; it first goes through the opt-in dispatch (cake_backend_opt_in) and then runs the backend’s default entry."budget": bound the step peak (forward through backward, outputs included) by the bytesBof a head that materializes the BF16 logits of the valid rows (B = 2 * T_v * V + 2 * V * H + 2 * T * H); the planner targetsBless a 2 % margin: per call, the fastest exact weight-gradient form whose predicted peak fits – the fused or the unfused cast, or the forward accumulating the leading vocabulary rows and the backward recomputing the others in vocabulary blocks (cake_backend_opt_in.memory_plan). Bitwise the sameloss/logp/dX/dWas"fast"; the recompute forms run more GEMM work. A default call whose automatic form is a budget form gives the same bits, except that a call with zero-weight policy rows decides its row split as"budget"does (dX/dWwithin the accuracy budget; see the package README). The bound assumes a backward that frees the graph (retain_graph=Truekeeps the forward’s state, also for a default call); its margin is measured on GB200, GB300 and B200 (see the package README)."keep_logits"belongs tocake_lm_head_det_train_logprob(aValueErrorhere). No environment variable selects the mode; the switches above (compact_rows,fuse_dw_cast,dx_finalize,hidden_count,gemm_tuning, and theirFLASHINFER_CAKE_LM_HEAD_LOSS_*variables) are inputs of the automatic form’s rule. The non-deterministic moduleflashinfer.cake_lm_head_nondet_train, whose entries refer to this description, has no automatic form: its defaultNoneis the fastest form.num_valid_rows (int, optional) – The number of labels
>= 0(not-100), as a host integer (a Python or NumPy integer or a CPU scalar tensor; a CUDA tensor is rejected). With compaction the forward then plans on it without waiting for queued device work: the valid-row index is formed on the device and the count is checked there by one asynchronous assertion – a wrong count ends the CUDA context with a device-side assertion. Exception: ansm_103acall whose library CUDA 13.0 built keeps the host-count schedule and reads the count back once (see the package README’s toolchain note).None(default): the forward counts the valid rows itself, which waits for the device once per call. Not derived fromloss_div.check_num_valid_rows (bool) – Read the count back and raise
ValueErrorwhennum_valid_rowsis wrong (one synchronization; for debugging).Truewithoutnum_valid_rowsis aValueError.
Notes
Fixed sequential chunk order, no atomics:
loss,logp,dXanddWare bitwise reproducible run to run. The det name is the mode, so there is nodeterministickeyword (passing one is aTypeError); torch’s deterministic mode (torch.use_deterministic_algorithms) changes nothing.Precision boundary: BF16 GEMM output promoted to FP32 for the max / log-sum-exp / loss arithmetic, BF16
dlogits, FP32dWaccumulation with one cast at the output, BF16dX(FP32 accumulation, one cast). Gradients are produced only for inputs that require them (a frozenXorWskips its GEMM). The saved accumulators are re-scaled, never mutated, so a retained graph may run the backward repeatedly.T == 0gives loss 0, an emptylogpand zero gradients. Under the backend’s default gradient placement (cake_backend.set_gradient_placement,"auto") a call no backward can follow (grad mode off, also the first pass of a reentranttorch.utils.checkpoint) computes the loss only, and inside a non-reentranttorch.utils.checkpointthe recomputation forms the gradient accumulators instead of the checkpointed forward (one gradient pass per step); every output is bitwise the same.A call with neither
memory_modenornum_valid_rowswhoseWrequires no gradient runs the backend’s default entry (cake_backend) after this function’s argument checks (check_num_valid_rows, thenmemory_modeandnum_valid_rowsagainstNone, thenW.requires_grad). Every other call goes throughcake_backend_opt_in: an explicitmemory_mode="fast"without a count, and a default call whose step forms no BF16dWon the cake backend on CUDA, then run the same default entry; a default call that can switch (grad mode on) runs this entry’s autograd function ofcake_backend_opt_in(its fast form bitwise the default entry’s); a default call under grad mode off whoseWrequires a gradient (e.g. the first pass of a reentranttorch.utils.checkpoint) runs the fast form’s forward with its hidden valid-row count checked against the step’s bound (the same bits); every other such call takes its opt-in forms. Withmemory_mode="budget", or a default call whose automatic form is a budget form, a backward that frees the graph releases the saved accumulators as soon as it has used them (a second backward then raises).cake_lm_head_det_train_preloadloads the libraries and creates the per-device state ahead of a process’s first call.Host side: inputs are validated and bound on the first call for an input binding (
data_ptr, shape, stride and dtype of every input plus the options); later calls with the same binding launch from the remembered argument plans (flashinfer.experimental.cake.cake_lm_head_det_train.cake_backend.BINDING_CACHE; the calls throughcake_backend_opt_in– the default loss calls that form a weight gradient among them, and a log-probability backward in a budget form – keep theirs incake_backend_opt_in.BINDING_CACHE).FLASHINFER_CAKE_LM_HEAD_LOSS_BINDING_CACHE=0disables both caches andFLASHINFER_CAKE_LM_HEAD_LOSS_BINDING_CACHE_CAPACITY(a positive integer; default 64) sets their capacity: both are read when this module is imported and applied throughBINDING_CACHE.configureon the first call that reaches each backend module; a latercake_backend.BINDING_CACHE.configure(...)sets that cache only. Peak temporary memory is reported separately from the weights, the outputs and the FP32 accumulators bycake_backend.memory_report.- Returns:
loss (torch.Tensor) – FP32 scalar.
logp (torch.Tensor) – Detached FP32
[T], only withreturn_logp=True.