Architecture
Grouped-Query Attention (GQA)
Grouped-query attention is a middle ground between multi-head and multi-query attention: query heads are split into groups, and each group shares one key head and one value head.
In depth
Grouped-query attention is a middle ground between multi-head and multi-query attention: query heads are split into groups, and each group shares one key head and one value head. With 8 KV heads serving 32 query heads, the KV cache shrinks by 4x with little quality loss. The GQA paper showed quality close to full multi-head attention at speeds close to MQA, and it is now standard in open models such as Llama, Mistral, and Qwen.
Primary source: arxiv.org
Example
The GQA paper showed quality close to full multi-head attention at speeds close to MQA, and it is now standard in open models such as Llama, Mistral, and Qwen.
Go deeper at Developers Digest
Hands-on guides, comparisons, and tutorials that cover Architecture.
FAQ
Common questions
What is Grouped-Query Attention (GQA)?
Grouped-query attention is a middle ground between multi-head and multi-query attention: query heads are split into groups, and each group shares one key head and one value head.
Why does Grouped-Query Attention (GQA) matter for AI developers?
Grouped-Query Attention (GQA) sits in the Architecture part of the AI stack. Understanding it helps you make better decisions when building, debugging, and shipping AI features.
Where can I learn more about Grouped-Query Attention (GQA)?
Developers Digest publishes tutorials and videos that cover Architecture topics including Grouped-Query Attention (GQA). Check the blog and YouTube channel for hands-on walkthroughs.
Related
Related terms
Architecture
Attention Sinks
Attention sinks are the first few tokens of a sequence, which LLMs learn to send large attention scores to even when those tokens carry no special meaning.
Architecture
FlashAttention
An exact attention algorithm that produces the same result as standard attention but reorganizes the work around GPU memory.
Architecture
PagedAttention
An attention algorithm that stores the KV cache in fixed-size blocks that do not need to be contiguous in GPU memory, borrowing the virtual memory and paging idea from operating systems.
Put this concept to work
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever