flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_logprob¶
- flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_logprob(X: Tensor, W: Tensor, labels: Tensor, *, chunk_size: int = 4096, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False, temperature: float | Tensor | None = None, check_temperature: bool = False) Tensor¶
Differentiable chunked selected-token log-probability, SM100 / SM103 / SM107 only.
Returns the FP32
logp [T](logp_t = z[t, y_t] - logsumexp_v z[t, v]for valid rows, 0 for rows whose label is-100) of the projectionz = X @ W^Tfor arbitrary downstream losses; the per-row gradientdlogparrives from autograd. The forward saves the FP32 log-sum-exp and the valid-row index; the backward recomputes each chunk’s logits (four GEMMs per chunk; the recompute writes the logit gradient from its epilogue) and produces BF16dXanddWthrough FP32 accumulators with one cast each (withfuse_dw_castthe last chunk’s weight-gradient GEMM writes the BF16dWfrom its epilogue). Arguments as incake_lm_head_det_train_loss(), including the switchescompact_rows,fuse_dw_cast,dx_finalize,dw_stream,hidden_countandgemm_tuningwith the same defaults (Nonereads the sameFLASHINFER_CAKE_LM_HEAD_LOSS_*variable; every switch is resolved once in the forward and the backward recomputes under the same values);dWis returned in BF16.memory_mode("budget": the backward’s weight-gradient form within the same bound; the defaultNone: the automatic form, decided in the backward, which keeps the default entry’s forward; a default call without a temperature, of at most two chunks of rows, whose valid rows fit one chunk keeps their BF16 logits, at most one chunk, from the forward to its backward, which forms the logit gradient from them instead of recomputing the logits, where the fast backward would run no zero-gradient row skip – not undertorch.utils.checkpoint, bitwise the same outputs;"fast"turns this off),num_valid_rowsandcheck_num_valid_rows(the forward’s count) as incake_lm_head_det_train_loss(). A default call whose backward runs a budget form runs it without the zero-gradient row skip: itsdX/dWbits can then differ from the fastest form’s where valid rows have a zero upstream gradient (within the accuracy budget, the same bits on every run).memory_mode="keep_logits"(this entry only; opt-in; withnum_valid_rowsits forward runs on the index formed from the caller’s count, without a readback): where a backward can follow (grad mode on,XorWtrainable) the forward keeps the BF16 logits of the valid rows,2 * T_v * Vbytes from the forward to the backward (the chunk bound of the memory rule does not hold for them), and the first backward forms the logit gradient from them instead of recomputing the logits (three GEMMs per chunk instead of four; a second backward through a retained graph recomputes); where none can follow (torch.no_grad,torch.inference_mode, frozen inputs) the call is the default one. Bitwise the samelogp,dXanddW. As forcake_lm_head_det_train_loss(), the det name is the mode: there is nodeterministickeyword (TypeError) and torch’s deterministic mode changes nothing.temperature(defaultNone, the same as1.0: no temperature) is a positive Python number or a float32[T]tensor onX’s device (one value per row, any stride); it is not differentiable (a tensor withrequires_grad=Trueis rejected). The result islogp_t = z[t, y_t] / tau_t - logsumexp_v(z[t, v] / tau_t)with the gradient
(1[v = y_t] - softmax(z_t / tau_t)_v) / tau_tper logit. The BF16 projectionzis unchanged; each logit is widened to FP32 and divided once (IEEE FP32 division), the upstreamdlogp_tis divided once per row, anddX/dWsee the temperature only through the logit gradient. A scalarsgives the same bits astorch.full((T,), s);Noneand1.0run the path without a temperature. Every value must be positive and finite, also on ignored rows (label-100). A bad tensor value stops the CUDA stream with a device-side assertion, which ends the CUDA context (no host synchronization);check_temperature=Truereads the check back once and raisesValueErrorinstead. Outputs are defined while|z / tau_t| < 3e38on every row.memory_mode="keep_logits"takes no temperature (ValueError).- Parameters:
X (torch.Tensor) – BF16
[T, H]hidden states, with the layout rules ofcake_lm_head_det_train_loss().W (torch.Tensor) – Contiguous BF16
[V, H]output weight.labels (torch.Tensor) – int64
[T]selected token ids;-100marks an ignored row.chunk_size (int, default=4096) – Maximum token rows per logits chunk.
backend (str, default="cake") – Only
"cake"is supported.compact_rows – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().fuse_dw_cast – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().dx_finalize – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().dw_stream – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().hidden_count – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().gemm_tuning – Execution switches with the contracts and defaults of
cake_lm_head_det_train_loss().memory_mode (str, optional) – Backward memory policy described above; this entry also accepts
"keep_logits".num_valid_rows (int, optional) – Caller-supplied host count of labels other than
-100.check_num_valid_rows (bool, default=False) – Read the valid-row count back and raise
ValueErroron a mismatch.temperature (float or torch.Tensor, optional) – Positive finite scalar or FP32
[T]temperature;Noneis equivalent to 1.0.check_temperature (bool, default=False) – Read tensor-temperature validation back and raise
ValueErrorinstead of a device assertion.