Inference
Decode Phase
The second phase of LLM inference, where the model generates output tokens one at a time, each step reading the KV cache built so far and adding to it.
In depth
The second phase of LLM inference, where the model generates output tokens one at a time, each step reading the KV cache built so far and adding to it. Because each request contributes only one new token per step, decode leaves GPU compute underused, so servers batch many requests together to keep the hardware busy. Its speed is what users feel as time per output token once a response starts streaming.
Primary source: arxiv.org
Example
In practice, developers reach for Decode Phase when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Decode Phase?
The second phase of LLM inference, where the model generates output tokens one at a time, each step reading the KV cache built so far and adding to it.
Why does Decode Phase matter for AI developers?
Decode Phase sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Decode Phase?
Developers Digest publishes tutorials and videos that cover Inference topics including Decode Phase. Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Disaggregated Serving
An LLM serving architecture that runs the prefill and decode phases on separate GPUs and passes the KV cache between them.
Inference
Input vs Output Token Pricing
The split pricing model where LLM APIs charge one rate per million tokens you send (input) and a higher rate per million tokens the model generates (output).
Inference
Tokens per Second
A throughput measure of how many output tokens a model or serving system produces each second.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever