flashinfer.cake_lm_head_nondet_train.cake_lm_head_nondet_train_loss

flashinfer.cake_lm_head_nondet_train.cake_lm_head_nondet_train_loss(X: Tensor, W: Tensor, labels: Tensor, *, objective: str = 'ce', loss_div: float | None = None, infer_logp: Tensor | None = None, loss_weights: Tensor | None = None, chunk_size: int = 4096, return_logp: bool = False, grad_weight_dtype: dtype = torch.bfloat16, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False)

Differentiable chunked LM-head projection + loss with a memory-bounded backward and a slab-summed dX reduction, SM100 / SM103 / SM107 only.

See flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss() for the computation, the arguments, the switches and their FLASHINFER_CAKE_LM_HEAD_LOSS_* fallbacks, memory_mode, num_valid_rows and the outputs, except as the module docstring notes (the dX reduction, the zero-weight row split and the one-pass call).

Notes

The same precision boundaries (BF16 logits and dlogits, FP32 loss arithmetic, FP32 dX / dW accumulation, one final cast each). A sliced dX GEMM stores each K slice into an FP32 slab, and a slab sum adds the slices in slice order and stores the sum into the FP32 dX accumulator, so dX repeats bit for bit run to run; loss, logp and dW keep their fixed order. dx_finalize then selects only the compacted dX output’s one-pass scatter (each chunk’s slab sum has already added its slices). The gradient placement is the backend’s own process setting (cake_backend.set_gradient_placement of flashinfer.experimental.cake.cake_lm_head_nondet_train, default "auto", the deterministic backend’s rule).

There is no deterministic keyword (passing one is a TypeError). While torch.use_deterministic_algorithms(True) is in effect the call still runs this backend: PyTorch’s alert is raised before a K-sliced dX reduction (RuntimeError; with warn_only=True a UserWarning, then the call runs on); there is no fallback to the deterministic entry, and every knob (a dX k_slices pin included) reaches the backend unchanged.

A call with neither memory_mode nor num_valid_rows runs the backend’s default entry (cake_backend) after this function’s argument checks (backend, then check_num_valid_rows); a call with either goes through cake_backend_opt_in.

Returns:

  • loss (torch.Tensor) – FP32 scalar.

  • logp (torch.Tensor) – Detached FP32 [T], only with return_logp=True.