flashinfer.cake_lm_head_nondet_train.cake_lm_head_nondet_train_loss¶
- flashinfer.cake_lm_head_nondet_train.cake_lm_head_nondet_train_loss(X: Tensor, W: Tensor, labels: Tensor, *, objective: str = 'ce', loss_div: float | None = None, infer_logp: Tensor | None = None, loss_weights: Tensor | None = None, chunk_size: int = 4096, return_logp: bool = False, grad_weight_dtype: dtype = torch.bfloat16, backend: str = 'cake', compact_rows: bool | None = None, fuse_dw_cast: bool | None = None, dx_finalize: bool | None = None, dw_stream: bool | str | None = None, hidden_count: bool | None = None, gemm_tuning: Dict[str, Dict[str, Any]] | None = None, memory_mode: str | None = None, num_valid_rows: int | None = None, check_num_valid_rows: bool = False)¶
Differentiable chunked LM-head projection + loss with a memory-bounded backward and a slab-summed dX reduction, SM100 / SM103 / SM107 only.
See
flashinfer.cake_lm_head_det_train.cake_lm_head_det_train_loss()for the computation, the arguments, the switches and theirFLASHINFER_CAKE_LM_HEAD_LOSS_*fallbacks,memory_mode,num_valid_rowsand the outputs, except as the module docstring notes (the dX reduction, the zero-weight row split and the one-pass call).Notes
The same precision boundaries (BF16 logits and
dlogits, FP32 loss arithmetic, FP32dX/dWaccumulation, one final cast each). A sliced dX GEMM stores each K slice into an FP32 slab, and a slab sum adds the slices in slice order and stores the sum into the FP32dXaccumulator, sodXrepeats bit for bit run to run;loss,logpanddWkeep their fixed order.dx_finalizethen selects only the compacteddXoutput’s one-pass scatter (each chunk’s slab sum has already added its slices). The gradient placement is the backend’s own process setting (cake_backend.set_gradient_placementofflashinfer.experimental.cake.cake_lm_head_nondet_train, default"auto", the deterministic backend’s rule).There is no
deterministickeyword (passing one is aTypeError). Whiletorch.use_deterministic_algorithms(True)is in effect the call still runs this backend: PyTorch’s alert is raised before a K-sliced dX reduction (RuntimeError; withwarn_only=TrueaUserWarning, then the call runs on); there is no fallback to the deterministic entry, and every knob (a dXk_slicespin included) reaches the backend unchanged.A call with neither
memory_modenornum_valid_rowsruns the backend’s default entry (cake_backend) after this function’s argument checks (backend, thencheck_num_valid_rows); a call with either goes throughcake_backend_opt_in.- Returns:
loss (torch.Tensor) – FP32 scalar.
logp (torch.Tensor) – Detached FP32
[T], only withreturn_logp=True.