Inference
Test-Time Compute
The computation a model spends while answering a prompt, as opposed to the compute spent training it.
In depth
The computation a model spends while answering a prompt, as opposed to the compute spent training it. Spending more of it, through longer reasoning, sampling several answers, or searching against a verifier, can raise accuracy without changing the weights. A 2024 Google DeepMind and Berkeley paper found that allocating test-time compute adaptively by prompt difficulty let a smaller model outperform one 14x larger on some problems. Reasoning models and thinking budgets are the product form of this idea.
Primary source: arxiv.org
Example
In practice, developers reach for Test-Time Compute when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Test-Time Compute?
The computation a model spends while answering a prompt, as opposed to the compute spent training it.
Why does Test-Time Compute matter for AI developers?
Test-Time Compute sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Test-Time Compute?
Developers Digest publishes tutorials and videos that cover Inference topics including Test-Time Compute. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Thinking Tokens
The tokens a reasoning model generates while working through a problem before it writes its final answer.
Inference
Top-K / Top-P Sampling
Two methods for controlling the randomness of model output during token generation.
Inference
Token Counting
Measuring how many tokens a prompt will use before you send it, usually through a provider endpoint that runs the model's own tokenizer.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever