flashinfer.sampling.log_softmax¶
- flashinfer.sampling.log_softmax(logits: Tensor, temperature: float | Tensor | None = None, *, enable_pdl: bool | None = None, batch_invariant: bool = False)¶
The log-softmax row of
logits / temperaturein fp32, with a batch-invariant dataflow: the row every deterministic sampling path forms before gathering a token’s log-probability (the top-k log-probabilities, the ranks, the full-row consumers), the same bits whatever the batch, its padding, the row’s position or the SM count, and run-to-run deterministic.The dataflow is the class
lse.row(flashinfer/batch_invariant/README.md): the row maximummand the sumsof theex2.approxweights as fixed trees over the vocabulary, thenlogp_j = ((x_j - m) * c - lg2(s)) * ln2withc = log2(e) / temperature;log_probs()is this row gathered at the token ids, andsoftmax()underbatch_invariant=Trueforms its probabilities from the samemands(p_j = e_j / s), so the three share the row’s statistics bit for bit. The fingerprint isflashinfer.batch_invariant.fingerprint("log_softmax", N=, dtype=, temperature=).- Parameters:
logits (torch.Tensor) – CUDA tensor of logits of shape
(..., vocab_size), dtype bfloat16, float16 or float32 (the 16-bit inputs are widened exactly).temperature (Optional[Union[torch.Tensor, float]]) –
None(no scaling), a positive scalar, or a tensor with one positive value per row (float16, bfloat16, float32 or float64; a float64 temperature is rounded to float32), of shapelogits.shape[:-1]or(rows,). A scalar is validated on the host; a tensor’s values are not validated on the device (a check would need a synchronisation and would break CUDA-graph capture): every value must be positive and finite, a0yields a NaN row, a negative or non-finite value an undefined row. Callers whose temperature tensors carry0for greedy rows (vLLM, SGLang) must substitute1.0there: the greedy log-probabilities are read from the unscaled row.enable_pdl (Optional[bool]) – Whether to launch with Programmatic Dependent Launch;
None(default) follows the device.batch_invariant (bool) –
Trueruns the batch-invariant classlse.row(the pinned contract: the fingerprint and the guarantees above).False(the default) runs the operator’s regular kernel for this entry, which today is the same class: a future release may route it to a faster kernel without the invariance contract, so callers that need the contract passTrueexplicitly.
- Returns:
logprobs – float32 tensor of shape
logits.shapewith the natural log-probabilities of every element.- Return type:
torch.Tensor
- Raises:
flashinfer.batch_invariant.BatchInvariantUnsupportedError – For inputs outside the class (dtype, device, architecture); the entry never falls back.