Inference
Next-Token Prediction
The training objective behind GPT-style language models: given a sequence of tokens, predict a probability distribution over the token that comes next.
In depth
The training objective behind GPT-style language models: given a sequence of tokens, predict a probability distribution over the token that comes next. Pre-training minimizes the cross-entropy loss of that prediction across a huge text corpus. The same mechanism produces output at inference time, one token at a time. OpenAI's GPT-4 technical report describes the model as a Transformer pre-trained to predict the next token in a document, with post-training then shaping its behavior.
Primary source: arxiv.org
Example
In practice, developers reach for Next-Token Prediction when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Next-Token Prediction?
The training objective behind GPT-style language models: given a sequence of tokens, predict a probability distribution over the token that comes next.
Why does Next-Token Prediction matter for AI developers?
Next-Token Prediction sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Next-Token Prediction?
Developers Digest publishes tutorials and videos that cover Inference topics including Next-Token Prediction. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Multi-Token Prediction (MTP)
A training objective where a language model learns to predict the next several tokens at each position instead of only the next one, using extra output heads on a shared model trunk.
Inference
Pretraining
The first and most expensive stage of building a language model, where it learns to predict the next token across a huge unlabeled corpus of web text, books, and code.
Inference
Continued Pretraining
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever