Inference
Multi-Query Attention (MQA)
Multi-query attention is a variant of multi-head attention where all query heads share a single key head and a single value head.
In depth
Multi-query attention is a variant of multi-head attention where all query heads share a single key head and a single value head. Proposed by Noam Shazeer in 2019, it cuts the size of the KV cache and the memory bandwidth needed to load it at every decoding step, which makes token generation much faster. The trade-off is some quality loss, which is why most current models use grouped-query attention as a compromise.
Primary source: arxiv.org
Example
In practice, developers reach for Multi-Query Attention (MQA) when they need the capability described above as part of an AI feature or workflow.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Inference.
FAQ
Common questions
What is Multi-Query Attention (MQA)?
Multi-query attention is a variant of multi-head attention where all query heads share a single key head and a single value head.
Why does Multi-Query Attention (MQA) matter for AI developers?
Multi-Query Attention (MQA) sits in the Inference part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Multi-Query Attention (MQA)?
Developers Digest publishes tutorials and videos that cover Inference topics including Multi-Query Attention (MQA). Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Inference
Multi-Token Prediction (MTP)
A training objective where a language model learns to predict the next several tokens at each position instead of only the next one, using extra output heads on a shared model trunk.
Inference
Multi-Head Attention
Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value projections, then concatenates their outputs and projects them back to the model dimension.
Inference
KV Cache
A store of the key and value tensors a transformer has already computed for earlier tokens, kept in memory so they can be reused at each new decoding step.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever