Inference
Continued Pretraining
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language.
In depth
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language. The 2020 paper Don't Stop Pretraining showed that this domain-adaptive second phase improves downstream results, even for strong general models. It sits between pretraining and post-training. For example, DeepSeekMath continued pretraining a coder model on 120B math tokens, and Cursor's Composer 2 came from a continued pretraining run on coding data before RL.
Primary source: arxiv.org
Example
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Continued Pretraining?
Running more next-token-prediction training on an already pretrained model using a large unlabeled corpus from a target domain, such as code, math, medicine, or a new language.
Why does Continued Pretraining matter for AI developers?
Continued Pretraining sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Continued Pretraining?
Developers Digest publishes tutorials and videos that cover Inference topics including Continued Pretraining. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Pretraining
The first and most expensive stage of building a language model, where it learns to predict the next token across a huge unlabeled corpus of web text, books, and code.
Inference
Speculative Decoding
An inference technique where a small, fast draft model proposes several tokens ahead and the large target model checks them all in one parallel forward pass.
Inference
Multi-Token Prediction (MTP)
A training objective where a language model learns to predict the next several tokens at each position instead of only the next one, using extra output heads on a shared model trunk.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever