flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss

flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss(X: Tensor, W: Tensor, labels: Tensor, *, objective: str = 'ce', loss_div: float | None = None, infer_logp: Tensor | None = None, loss_weights: Tensor | None = None, chunk_size: int = 4096, return_logp: bool = False, grad_weight_dtype: dtype = torch.bfloat16, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False)

Differentiable chunked LM-head projection + loss with a memory-bounded backward, SM100 / SM103 / SM107 only.

Computes the projection z = X @ W^T over token chunks of chunk_size rows, the per-row log-probability of the selected token logp_t = z[t, y_t] - logsumexp_v z[t, v] and one of two objectives, without ever materializing a vocabulary-sized tensor of more than chunk_size rows. The forward already produces the FP32 gradient accumulators of the trainable inputs (three GEMMs per chunk, no logits recomputation); the backward applies the incoming scalar gradient and performs the single cast to the output dtypes.

Parameters:
  • X (torch.Tensor) – BF16 [T, H] hidden states, row-major with any leading stride (H a multiple of 256; a stride whose byte pitch is not a multiple of 16 is copied to a contiguous tensor first).

  • W (torch.Tensor) – BF16 [V, H] output weight, contiguous (V a multiple of 256).

  • labels (torch.Tensor) – int64 [T]; -100 marks an ignored row (zero loss, zero gradient, logp = 0).

  • objective (str) – "ce": loss = -sum(logp[valid]) / loss_div. "policy": loss = -sum(loss_weights * min(exp(logp - infer_logp), 2)) over valid rows; the per-row logit-gradient scale is -loss_weights_t * ratio_t when ratio_t <= 2 and 0 above the clipping boundary.

  • loss_div (Optional[float]) – Positive scalar divisor of the cross-entropy objective (caller-supplied, never replaced by a local token mean). Required for "ce".

  • infer_logp (Optional[torch.Tensor]) – FP32 [T] inference log-probabilities of the policy objective.

  • loss_weights (Optional[torch.Tensor]) – FP32 [T] signed weights of the policy objective; they already include masking and normalization (ignored rows carry weight 0).

  • chunk_size (int) – Token chunk C (default 4096). Every logits / dlogits / probability buffer spans at most C tokens; a batch smaller than C is one chunk, and a tail T % C is neither dropped nor padded. Changing C may change reduction rounding, never the loss definition.

  • return_logp (bool) – Also return the detached FP32 logp [T] (a diagnostic output).

  • grad_weight_dtype (torch.dtype) – dtype of the weight gradient: torch.bfloat16 (default) or torch.float32. dW is accumulated in FP32 across chunks and cast once at the output boundary. Through this autograd entry the value must equal W.dtype (the autograd engine casts every gradient to its leaf’s dtype); an FP32 dW for a BF16 W is available from the explicit cake_backend.forward_loss / backward_loss pair.

  • backend (str) – Only "cake".

  • compact_rows (bool, optional) – Chunk over the valid rows only (rows whose label is not -100): the valid-row index is formed once per call, each chunk’s rows of X are gathered into one reusable [chunk_size, H] buffer and logp / dX are scattered back with exact zeros on the ignored rows – the same computation over fewer rows (per-row logp is bitwise that of the uncompacted path; the loss / dW reductions run over different chunk boundaries and differ by FP32 rounding). None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_COMPACT_ROWS (on unless set to 0; the backend’s own default is True). Compaction costs one device synchronization per call, the valid-row count (the log-probability backward reuses the forward’s index).

  • fuse_dw_cast (bool, optional) – Run the last chunk’s weight-gradient GEMM in the backward, where the upstream scalar gradient is known, with the scale and the output cast fused into its epilogue: the forward accumulates the chunks before it in FP32 (no accumulator at all for a one-chunk call), the backward writes dW = cast(g * (dW_acc + dz_c^T @ X_c)) straight from the GEMM and the separate pass over the [V, H] accumulator disappears. The same FP32 operations in the same order: dW is bitwise the unfused result. The last chunk’s BF16 dlogits rows and a view of its rows of X are saved for the backward (an in-place write to X between the forward and the backward raises PyTorch’s saved-tensor version error). None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_FUSE_DW_CAST (on unless set to 0; the backend’s own default is True).

  • dx_finalize (bool, optional) – Add the K-slice slabs of a sliced dX GEMM with one fixed-order kernel and, with compact_rows, write the [T, H] dX in one pass (bf16(g * dX_acc) on the valid rows, exact zeros elsewhere) instead of casting, zero-filling and scattering the compact rows; the same FP32 operations in the same order, so dX is bitwise the same. None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_DX_FINALIZE (on unless set to 0; the backend’s own default is True).

  • dw_stream (bool or str, optional) – Launch each chunk’s weight-gradient accumulate GEMM on a per-device side stream, forked after the chunk’s row gradients and joined before the chunk buffer is reused and before the call returns, so it overlaps the tail of the chunk’s dX GEMM: "auto" for calls of three or more chunks, True for every multi-chunk call, False never; the same kernels in the same order per kernel, so every output is bitwise the same. None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_DW_STREAM (0 / false / off / no -> False, 1 / true / on / yes -> True, anything else -> "auto"; the backend’s own default is "auto").

  • hidden_count (bool, optional) – Form a compacted call’s valid-row index on the device and queue chunk 0’s row gather and logits GEMM (its device-count form, bounded by the count read from device memory) before the count reaches the host through a pinned cell and a CUDA event, instead of counting and indexing on the host before the first launch; the same kernels per row, so every output is bitwise the same. None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_HIDDEN_COUNT (on unless set to 0; the backend’s own default is True).

  • gemm_tuning (dict, optional) – Explicit per-GEMM knobs, {"logits": {...}, "dx": {...}, "dw": {...}} (e.g. {"dx": {"tile_n": 256, "k_slices": 2}}; a knob given as None pins the default form), winning over the per-chunk rules for the knobs they name; a form the compiled program does not register fails closed. None (default) reads FLASHINFER_CAKE_LM_HEAD_LOSS_GEMM_TUNING as a JSON object (unset or blank: no explicit knob, the backend’s own default; invalid JSON is a ValueError).

  • memory_mode (str, optional) – None (the default): the automatic form. A call that forms a BF16 dW on CUDA (grad mode on, W requiring a gradient) runs the fastest form while its predicted step peak, with that form’s own buffer lifetimes, stays within B less the 2 % margin below, else the "budget" form the planner picks when one fits, else the fastest form (cake_backend_opt_in.auto_memory_plan). A call whose valid rows fit one chunk keeps the fastest form: with the fused weight-gradient cast (fuse_dw_cast, on by default) it holds no FP32 [V, H] accumulator (without it, it does), and its step peak can exceed B (see the package README). It plans on the valid rows of the forward’s count or of num_valid_rows; an uncompacted call without num_valid_rows keeps the fastest form, as does every call without such a dW. "fast": always the fastest form, whose FP32 [V, H] weight-gradient accumulator lives next to the chunk buffers; it first goes through the opt-in dispatch (cake_backend_opt_in) and then runs the backend’s default entry. "budget": bound the step peak (forward through backward, outputs included) by the bytes B of a head that materializes the BF16 logits of the valid rows (B = 2 * T_v * V + 2 * V * H + 2 * T * H); the planner targets B less a 2 % margin: per call, the fastest exact weight-gradient form whose predicted peak fits – the fused or the unfused cast, or the forward accumulating the leading vocabulary rows and the backward recomputing the others in vocabulary blocks (cake_backend_opt_in.memory_plan). Bitwise the same loss / logp / dX / dW as "fast"; the recompute forms run more GEMM work. A default call whose automatic form is a budget form gives the same bits, except that a call with zero-weight policy rows decides its row split as "budget" does (dX / dW within the accuracy budget; see the package README). The bound assumes a backward that frees the graph (retain_graph=True keeps the forward’s state, also for a default call); its margin is measured on GB200, GB300 and B200 (see the package README). "keep_logits" belongs to cake_lm_head_det_train_logprob (a ValueError here). No environment variable selects the mode; the switches above (compact_rows, fuse_dw_cast, dx_finalize, hidden_count, gemm_tuning, and their FLASHINFER_CAKE_LM_HEAD_LOSS_* variables) are inputs of the automatic form’s rule. The non-deterministic module flashinfer.cake_lm_head_nondet_train, whose entries refer to this description, has no automatic form: its default None is the fastest form.

  • num_valid_rows (int, optional) – The number of labels >= 0 (not -100), as a host integer (a Python or NumPy integer or a CPU scalar tensor; a CUDA tensor is rejected). With compaction the forward then plans on it without waiting for queued device work: the valid-row index is formed on the device and the count is checked there by one asynchronous assertion – a wrong count ends the CUDA context with a device-side assertion. Exception: an sm_103a call whose library CUDA 13.0 built keeps the host-count schedule and reads the count back once (see the package README’s toolchain note). None (default): the forward counts the valid rows itself, which waits for the device once per call. Not derived from loss_div.

  • check_num_valid_rows (bool) – Read the count back and raise ValueError when num_valid_rows is wrong (one synchronization; for debugging). True without num_valid_rows is a ValueError.

Notes

Fixed sequential chunk order, no atomics: loss, logp, dX and dW are bitwise reproducible run to run. The det name is the mode, so there is no deterministic keyword (passing one is a TypeError); torch’s deterministic mode (torch.use_deterministic_algorithms) changes nothing.

Precision boundary: BF16 GEMM output promoted to FP32 for the max / log-sum-exp / loss arithmetic, BF16 dlogits, FP32 dW accumulation with one cast at the output, BF16 dX (FP32 accumulation, one cast). Gradients are produced only for inputs that require them (a frozen X or W skips its GEMM). The saved accumulators are re-scaled, never mutated, so a retained graph may run the backward repeatedly. T == 0 gives loss 0, an empty logp and zero gradients. Under the backend’s default gradient placement (cake_backend.set_gradient_placement, "auto") a call no backward can follow (grad mode off, also the first pass of a reentrant torch.utils.checkpoint) computes the loss only, and inside a non-reentrant torch.utils.checkpoint the recomputation forms the gradient accumulators instead of the checkpointed forward (one gradient pass per step); every output is bitwise the same.

A call with neither memory_mode nor num_valid_rows whose W requires no gradient runs the backend’s default entry (cake_backend) after this function’s argument checks (check_num_valid_rows, then memory_mode and num_valid_rows against None, then W.requires_grad). Every other call goes through cake_backend_opt_in: an explicit memory_mode="fast" without a count, and a default call whose step forms no BF16 dW on the cake backend on CUDA, then run the same default entry; a default call that can switch (grad mode on) runs this entry’s autograd function of cake_backend_opt_in (its fast form bitwise the default entry’s); a default call under grad mode off whose W requires a gradient (e.g. the first pass of a reentrant torch.utils.checkpoint) runs the fast form’s forward with its hidden valid-row count checked against the step’s bound (the same bits); every other such call takes its opt-in forms. With memory_mode="budget", or a default call whose automatic form is a budget form, a backward that frees the graph releases the saved accumulators as soon as it has used them (a second backward then raises).

cake_lm_head_det_train_preload loads the libraries and creates the per-device state ahead of a process’s first call.

Host side: inputs are validated and bound on the first call for an input binding (data_ptr, shape, stride and dtype of every input plus the options); later calls with the same binding launch from the remembered argument plans (flashinfer.experimental.cake.cake_lm_head_det_train.cake_backend.BINDING_CACHE; the calls through cake_backend_opt_in – the default loss calls that form a weight gradient among them, and a log-probability backward in a budget form – keep theirs in cake_backend_opt_in.BINDING_CACHE). FLASHINFER_CAKE_LM_HEAD_LOSS_BINDING_CACHE=0 disables both caches and FLASHINFER_CAKE_LM_HEAD_LOSS_BINDING_CACHE_CAPACITY (a positive integer; default 64) sets their capacity: both are read when this module is imported and applied through BINDING_CACHE.configure on the first call that reaches each backend module; a later cake_backend.BINDING_CACHE.configure(...) sets that cache only. Peak temporary memory is reported separately from the weights, the outputs and the FP32 accumulators by cake_backend.memory_report.

Returns:

  • loss (torch.Tensor) – FP32 scalar.

  • logp (torch.Tensor) – Detached FP32 [T], only with return_logp=True.