flashinfer.sampling.log_probs¶
- flashinfer.sampling.log_probs(logits: Tensor, token_ids: Tensor, temperature: float | Tensor | None = None, *, return_lse: bool = False, enable_pdl: bool | None = None, batch_invariant: bool = False)¶
The log-probability of one token per row under
softmax(logits / temperature), with a batch-invariant dataflow: the teacher-forced score of a sampled token (a trainer re-scoring a rollout) and the returned log-probability of a decode step (an inference engine) are the same bits whatever the batch, its padding, the row’s position or the SM count, and run-to-run deterministic.The dataflow is the class
lse.row(flashinfer/batch_invariant/README.md): the row maximummand the sumsof theex2.approxweights as fixed trees over the vocabulary, thenlogp = ((x_t - m) * c - lg2(s)) * ln2withc = log2(e) / temperature;softmax()underbatch_invariant=Truecomputes its probabilities from the samemands. The fingerprint isflashinfer.batch_invariant.fingerprint("log_probs", N=, dtype=, temperature=).- Parameters:
logits (torch.Tensor) – CUDA tensor of logits of shape
(..., vocab_size), dtype bfloat16, float16 or float32 (the 16-bit inputs are widened exactly).token_ids (torch.Tensor) – Integer tensor of shape
logits.shape[:-1](one token id scored per row) orlogits.shape[:-1] + (K,)(Kids per row: a sampled token with its top-k or requested alternatives, in one launch). An id outside[0, vocab_size)gives NaN.temperature (Optional[Union[torch.Tensor, float]]) –
None(no scaling), a positive scalar, or a tensor with one positive value per row (float16, bfloat16, float32 or float64; a float64 temperature is rounded to float32), of shapelogits.shape[:-1]or(rows,). A scalar is validated on the host; a tensor’s values are not validated on the device (a check would need a synchronisation and would break CUDA-graph capture): every value must be positive and finite, a0yields a NaN row, a negative or non-finite value an undefined row. Callers whose temperature tensors carry0for greedy rows (vLLM, SGLang) must substitute1.0there: the greedy log-probabilities are read from the unscaled row.return_lse (bool) – Also return the natural log-sum-exp of each scaled row (the normaliser), from the same launch.
enable_pdl (Optional[bool]) – Whether to launch with Programmatic Dependent Launch;
None(default) follows the device.batch_invariant (bool) –
Trueruns the batch-invariant classlse.row(the pinned contract: the fingerprint and the guarantees above).False(the default) runs the operator’s regular kernel for this entry, which today is the same class: a future release may route it to a faster kernel without the invariance contract, so callers that need the contract passTrueexplicitly.
- Returns:
logp (torch.Tensor) – float32 tensor of the token ids’ shape with the natural log-probabilities.
lse (torch.Tensor) – Only with
return_lse=True: float32 tensor of shapelogits.shape[:-1]with the natural log-sum-exp of each scaled row.
Notes
For two callers to agree bit for bit (a sampler and the trainer re-scoring its tokens) they must call with the same arguments: raw logits with the temperature (the class scales inside its chain) or identically pre-scaled logits with
temperature=None. A row view into a wider tensor whose row stride is a multiple of the vector alignment (16 bytes) is read in place; other views are copied first.- Raises:
flashinfer.batch_invariant.BatchInvariantUnsupportedError – For inputs outside the class (dtype, device, architecture); the entry never falls back.
Examples
>>> import torch >>> import flashinfer >>> logits = torch.randn(4, 32000, device="cuda", dtype=torch.bfloat16) >>> token_ids = torch.randint(0, 32000, (4,), device="cuda") >>> logp = flashinfer.sampling.log_probs(logits, token_ids, temperature=0.7) >>> logp.shape, logp.dtype (torch.Size([4]), torch.float32)