flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_logprob

flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_logprob(X: Tensor, W: Tensor, labels: Tensor, *, chunk_size: int = 4096, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False, temperature: float | Tensor | None = None, check_temperature: bool = False) → Tensor

Differentiable chunked selected-token log-probability, SM100 / SM103 / SM107 only.

Returns the FP32 logp [T] (logp_t = z[t, y_t] - logsumexp_v z[t, v] for valid rows, 0 for rows whose label is -100) of the projection z = X @ W^T for arbitrary downstream losses; the per-row gradient dlogp arrives from autograd. The forward saves the FP32 log-sum-exp and the valid-row index; the backward recomputes each chunk’s logits (four GEMMs per chunk; the recompute writes the logit gradient from its epilogue) and produces BF16 dX and dW through FP32 accumulators with one cast each (with fuse_dw_cast the last chunk’s weight-gradient GEMM writes the BF16 dW from its epilogue). Arguments as in cake_lm_head_det_train_loss(), including the switches compact_rows, fuse_dw_cast, dx_finalize, dw_stream, hidden_count and gemm_tuning with the same defaults (None reads the same FLASHINFER_CAKE_LM_HEAD_LOSS_* variable; every switch is resolved once in the forward and the backward recomputes under the same values); dW is returned in BF16. memory_mode ("budget": the backward’s weight-gradient form within the same bound; the default None: the automatic form, decided in the backward, which keeps the default entry’s forward; a default call without a temperature, of at most two chunks of rows, whose valid rows fit one chunk keeps their BF16 logits, at most one chunk, from the forward to its backward, which forms the logit gradient from them instead of recomputing the logits, where the fast backward would run no zero-gradient row skip – not under torch.utils.checkpoint, bitwise the same outputs; "fast" turns this off), num_valid_rows and check_num_valid_rows (the forward’s count) as in cake_lm_head_det_train_loss(). A default call whose backward runs a budget form runs it without the zero-gradient row skip: its dX / dW bits can then differ from the fastest form’s where valid rows have a zero upstream gradient (within the accuracy budget, the same bits on every run). memory_mode="keep_logits" (this entry only; opt-in; with num_valid_rows its forward runs on the index formed from the caller’s count, without a readback): where a backward can follow (grad mode on, X or W trainable) the forward keeps the BF16 logits of the valid rows, 2 * T_v * V bytes from the forward to the backward (the chunk bound of the memory rule does not hold for them), and the first backward forms the logit gradient from them instead of recomputing the logits (three GEMMs per chunk instead of four; a second backward through a retained graph recomputes); where none can follow (torch.no_grad, torch.inference_mode, frozen inputs) the call is the default one. Bitwise the same logp, dX and dW. As for cake_lm_head_det_train_loss(), the det name is the mode: there is no deterministic keyword (TypeError) and torch’s deterministic mode changes nothing.

temperature (default None, the same as 1.0: no temperature) is a positive Python number or a float32 [T] tensor on X’s device (one value per row, any stride); it is not differentiable (a tensor with requires_grad=True is rejected). The result is

logp_t = z[t, y_t] / tau_t - logsumexp_v(z[t, v] / tau_t)

with the gradient (1[v = y_t] - softmax(z_t / tau_t)_v) / tau_t per logit. The BF16 projection z is unchanged; each logit is widened to FP32 and divided once (IEEE FP32 division), the upstream dlogp_t is divided once per row, and dX / dW see the temperature only through the logit gradient. A scalar s gives the same bits as torch.full((T,), s); None and 1.0 run the path without a temperature. Every value must be positive and finite, also on ignored rows (label -100). A bad tensor value stops the CUDA stream with a device-side assertion, which ends the CUDA context (no host synchronization); check_temperature=True reads the check back once and raises ValueError instead. Outputs are defined while |z / tau_t| < 3e38 on every row. memory_mode="keep_logits" takes no temperature (ValueError).

Parameters:
  • X (torch.Tensor) – BF16 [T, H] hidden states, with the layout rules of cake_lm_head_det_train_loss().

  • W (torch.Tensor) – Contiguous BF16 [V, H] output weight.

  • labels (torch.Tensor) – int64 [T] selected token ids; -100 marks an ignored row.

  • chunk_size (int, default=4096) – Maximum token rows per logits chunk.

  • backend (str, default="cake") – Only "cake" is supported.

  • compact_rows – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • fuse_dw_cast – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • dx_finalize – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • dw_stream – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • hidden_count – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • gemm_tuning – Execution switches with the contracts and defaults of cake_lm_head_det_train_loss().

  • memory_mode (str, optional) – Backward memory policy described above; this entry also accepts "keep_logits".

  • num_valid_rows (int, optional) – Caller-supplied host count of labels other than -100.

  • check_num_valid_rows (bool, default=False) – Read the valid-row count back and raise ValueError on a mismatch.

  • temperature (float or torch.Tensor, optional) – Positive finite scalar or FP32 [T] temperature; None is equivalent to 1.0.

  • check_temperature (bool, default=False) – Read tensor-temperature validation back and raise ValueError instead of a device assertion.