flashinfer.sampling.log_probs

flashinfer.sampling.log_probs(logits: Tensor, token_ids: Tensor, temperature: float | Tensor | None = None, *, return_lse: bool = False, enable_pdl: bool | None = None, batch_invariant: bool = False)

The log-probability of one token per row under softmax(logits / temperature), with a batch-invariant dataflow: the teacher-forced score of a sampled token (a trainer re-scoring a rollout) and the returned log-probability of a decode step (an inference engine) are the same bits whatever the batch, its padding, the row’s position or the SM count, and run-to-run deterministic.

The dataflow is the class lse.row (flashinfer/batch_invariant/README.md): the row maximum m and the sum s of the ex2.approx weights as fixed trees over the vocabulary, then logp = ((x_t - m) * c - lg2(s)) * ln2 with c = log2(e) / temperature; softmax() under batch_invariant=True computes its probabilities from the same m and s. The fingerprint is flashinfer.batch_invariant.fingerprint("log_probs", N=, dtype=, temperature=).

Parameters:
  • logits (torch.Tensor) – CUDA tensor of logits of shape (..., vocab_size), dtype bfloat16, float16 or float32 (the 16-bit inputs are widened exactly).

  • token_ids (torch.Tensor) – Integer tensor of shape logits.shape[:-1] (one token id scored per row) or logits.shape[:-1] + (K,) (K ids per row: a sampled token with its top-k or requested alternatives, in one launch). An id outside [0, vocab_size) gives NaN.

  • temperature (Optional[Union[torch.Tensor, float]]) – None (no scaling), a positive scalar, or a tensor with one positive value per row (float16, bfloat16, float32 or float64; a float64 temperature is rounded to float32), of shape logits.shape[:-1] or (rows,). A scalar is validated on the host; a tensor’s values are not validated on the device (a check would need a synchronisation and would break CUDA-graph capture): every value must be positive and finite, a 0 yields a NaN row, a negative or non-finite value an undefined row. Callers whose temperature tensors carry 0 for greedy rows (vLLM, SGLang) must substitute 1.0 there: the greedy log-probabilities are read from the unscaled row.

  • return_lse (bool) – Also return the natural log-sum-exp of each scaled row (the normaliser), from the same launch.

  • enable_pdl (Optional[bool]) – Whether to launch with Programmatic Dependent Launch; None (default) follows the device.

  • batch_invariant (bool) – True runs the batch-invariant class lse.row (the pinned contract: the fingerprint and the guarantees above). False (the default) runs the operator’s regular kernel for this entry, which today is the same class: a future release may route it to a faster kernel without the invariance contract, so callers that need the contract pass True explicitly.

Returns:

  • logp (torch.Tensor) – float32 tensor of the token ids’ shape with the natural log-probabilities.

  • lse (torch.Tensor) – Only with return_lse=True: float32 tensor of shape logits.shape[:-1] with the natural log-sum-exp of each scaled row.

Notes

For two callers to agree bit for bit (a sampler and the trainer re-scoring its tokens) they must call with the same arguments: raw logits with the temperature (the class scales inside its chain) or identically pre-scaled logits with temperature=None. A row view into a wider tensor whose row stride is a multiple of the vector alignment (16 bytes) is read in place; other views are copied first.

Raises:

flashinfer.batch_invariant.BatchInvariantUnsupportedError – For inputs outside the class (dtype, device, architecture); the entry never falls back.

Examples

>>> import torch
>>> import flashinfer
>>> logits = torch.randn(4, 32000, device="cuda", dtype=torch.bfloat16)
>>> token_ids = torch.randint(0, 32000, (4,), device="cuda")
>>> logp = flashinfer.sampling.log_probs(logits, token_ids, temperature=0.7)
>>> logp.shape, logp.dtype
(torch.Size([4]), torch.float32)