Inference
LLM Nondeterminism
The tendency of a language model endpoint to return different outputs for the same prompt, even at temperature 0.
In depth
The tendency of a language model endpoint to return different outputs for the same prompt, even at temperature 0. Thinking Machines Lab traced most of it to inference kernels that are not batch invariant: floating-point addition is not associative, so a request's numbers shift slightly depending on how many other requests share its batch, and batch size changes with server load. In practice, treat model output as a distribution, run evals more than once, and do not assume temperature 0 means repeatable.
Primary source: thinkingmachines.ai
Example
In practice, developers reach for LLM Nondeterminism when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is LLM Nondeterminism?
The tendency of a language model endpoint to return different outputs for the same prompt, even at temperature 0.
Why does LLM Nondeterminism matter for AI developers?
LLM Nondeterminism sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about LLM Nondeterminism?
Developers Digest publishes tutorials and videos that cover Inference topics including LLM Nondeterminism. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Logits
The raw, unnormalized scores a language model's output layer assigns to every token in its vocabulary at each step, before softmax turns them into probabilities.
Inference
Speculative Decoding
An inference technique where a small, fast draft model proposes several tokens ahead and the large target model checks them all in one parallel forward pass.
Inference
Top-K / Top-P Sampling
Two methods for controlling the randomness of model output during token generation.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever