Inference
YaRN
YaRN (Yet another RoPE extensioN method) is a technique for extending the context window of RoPE-based models beyond the length they were pretrained on.
In depth
YaRN (Yet another RoPE extensioN method) is a technique for extending the context window of RoPE-based models beyond the length they were pretrained on. It combines "NTK-by-parts" interpolation, which stretches the low-frequency rotary dimensions while leaving high-frequency ones alone, with a temperature adjustment on the attention logits. The paper reports needing 10x fewer tokens and 2.5x fewer training steps than earlier methods, and open-weight families such as Qwen document it as the way to run past their native context length.
Primary source: arxiv.org
Example
The paper reports needing 10x fewer tokens and 2.5x fewer training steps than earlier methods, and open-weight families such as Qwen document it as the way to run past their native context length.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is YaRN?
YaRN (Yet another RoPE extensioN method) is a technique for extending the context window of RoPE-based models beyond the length they were pretrained on.
Why does YaRN matter for AI developers?
YaRN sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about YaRN?
Developers Digest publishes tutorials and videos that cover Inference topics including YaRN. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
In-Context Learning
The ability of a language model to learn new tasks from examples or instructions provided in the prompt, without any weight updates or training.
Inference
Thinking Tokens
The tokens a reasoning model generates while working through a problem before it writes its final answer.
Inference
Context Rot
The decline in LLM output quality as the number of input tokens grows, even on simple tasks that fit well inside the context window.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever