flashinfer.sampling.log_softmax

flashinfer.sampling.log_softmax(logits: Tensor, temperature: float | Tensor | None = None, *, enable_pdl: bool | None = None, batch_invariant: bool = False)

The log-softmax row of logits / temperature in fp32, with a batch-invariant dataflow: the row every deterministic sampling path forms before gathering a token’s log-probability (the top-k log-probabilities, the ranks, the full-row consumers), the same bits whatever the batch, its padding, the row’s position or the SM count, and run-to-run deterministic.

The dataflow is the class lse.row (flashinfer/batch_invariant/README.md): the row maximum m and the sum s of the ex2.approx weights as fixed trees over the vocabulary, then logp_j = ((x_j - m) * c - lg2(s)) * ln2 with c = log2(e) / temperature; log_probs() is this row gathered at the token ids, and softmax() under batch_invariant=True forms its probabilities from the same m and s (p_j = e_j / s), so the three share the row’s statistics bit for bit. The fingerprint is flashinfer.batch_invariant.fingerprint("log_softmax", N=, dtype=, temperature=).

Parameters:
  • logits (torch.Tensor) – CUDA tensor of logits of shape (..., vocab_size), dtype bfloat16, float16 or float32 (the 16-bit inputs are widened exactly).

  • temperature (Optional[Union[torch.Tensor, float]]) – None (no scaling), a positive scalar, or a tensor with one positive value per row (float16, bfloat16, float32 or float64; a float64 temperature is rounded to float32), of shape logits.shape[:-1] or (rows,). A scalar is validated on the host; a tensor’s values are not validated on the device (a check would need a synchronisation and would break CUDA-graph capture): every value must be positive and finite, a 0 yields a NaN row, a negative or non-finite value an undefined row. Callers whose temperature tensors carry 0 for greedy rows (vLLM, SGLang) must substitute 1.0 there: the greedy log-probabilities are read from the unscaled row.

  • enable_pdl (Optional[bool]) – Whether to launch with Programmatic Dependent Launch; None (default) follows the device.

  • batch_invariant (bool) – True runs the batch-invariant class lse.row (the pinned contract: the fingerprint and the guarantees above). False (the default) runs the operator’s regular kernel for this entry, which today is the same class: a future release may route it to a faster kernel without the invariance contract, so callers that need the contract pass True explicitly.

Returns:

logprobs – float32 tensor of shape logits.shape with the natural log-probabilities of every element.

Return type:

torch.Tensor

Raises:

flashinfer.batch_invariant.BatchInvariantUnsupportedError – For inputs outside the class (dtype, device, architecture); the entry never falls back.