Kimi Linear: An Attention Architecture That Outperforms Full Attention

For years, linear attention has been the architecture that should have worked. Replace the quadratic self-attention in transformers with something that scales linearly with sequence length, and you get faster inference, longer contexts, and cheaper deployment. The problem has always been quality - linear attention variants consistently underperform full attention on reasoning, retrieval, and long-context tasks.

Kimi Linear, a technical report from Moonshot AI's Kimi team published on arXiv in October 2025 and resurging on Hacker News this week, claims to have broken that tradeoff. For the first time under fair comparisons, a hybrid linear attention architecture outperforms full attention across short-context, long-context, and reinforcement learning scaling regimes. The results are significant enough that Kimi K3 - Moonshot's 2.8 trillion parameter frontier model released just yesterday - is built on the same Kimi Delta Attention (KDA) mechanism.

What Kimi Linear Actually Does#

The paper introduces Kimi Delta Attention (KDA), a linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism. The key insight is about memory: linear attention functions as a finite-state RNN where the hidden state is the limited resource. KDA's improved gating makes more effective use of that limited state, letting the model retain and recall information across very long sequences without the quadratic cost of full attention.

The team pretrained a 3 billion activated parameter model with 48 billion total parameters, using a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA) - the same attention mechanism DeepSeek uses. Under an identical training recipe, Kimi Linear outperformed full MLA across every evaluated task while achieving:

• 75% reduction in KV cache memory
• Up to 6x decoding throughput at 1 million token context
• Better results on both short-context and long-context benchmarks

What makes this practically interesting is the hardware efficiency. The bespoke chunkwise algorithm uses a specialized variant of Diagonal-Plus-Low-Rank (DPLR) transition matrices. This substantially reduces computation compared to general DPLR formulations while staying consistent with the classical delta rule. In plain terms: it runs fast on GPUs without exotic kernels.

The team open-sourced the KDA kernel, vLLM implementations, and released both pre-trained and instruction-tuned model checkpoints on Hugging Face. You can run the 3B/48B model today on commodity hardware.

What HN Is Saying#

The Hacker News thread on Kimi Linear (145 points, 52 comments at time of writing) is notable not for disagreement about the results, but for the conversation it triggered about how this architecture feeds into the broader open-weights AI landscape.

The top-voted thread called out that the paper is 9 months old but newly relevant because Kimi K3 "has 69 KDA layers (the rest are 24 Gated MLA)" - meaning Moonshot validated the architecture at massive scale, not just in the 3B/48B demo model. HN user senko noted that "if you read the recently-released Kimi K3 paper, you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements)."

One point of technical debate: whether K3 uses the exact same KDA mechanism as the Kimi Linear paper. User yorwba questioned this, saying "it's not the same KDA as used in Kimi Linear," while GaggiX countered that both papers call it Kimi Delta Attention. The K3 paper itself mentions "Kimi Delta Attention and Attention Residuals, which improve information flow across sequence length and model depth" - so the core mechanism is shared with additions.

User bratao shared hands-on experience: "I started creating internal models using it, then the Gated Deltanet 2 came out, and it seems like an evolution of it in expressiveness. And in our tests it is really better than." This suggests the research direction is active and improving.

The most pointed question came from imrozim: "Any one knows how this holds up on long context retrieval (needle in a haystack, ruler) vs same size full attention model? Efficiency gains look great but that usually where linear attention hybrids fall apart." This is the open question for KDA - the paper shows strong benchmarks, but real-world needle-in-haystack tasks have historically been the Achilles heel of linear attention.

The thread also hosted a broader discussion about whether Moonshot's success comes from distillation of frontier models or genuine architectural innovation. User delichon warned: "If you want to believe that the success of Kimi is about distillation attacks, ignore this." The distillation debate surfaced because Anthropic's recent open-weights position statement had just hit the front page with 1,095 points, making the comparison top-of-mind.

Why It Matters#

Kimi Linear matters for three reasons that go beyond the numbers in the paper.

First, it validates a research direction that many in the field had started to write off. Linear attention has been "almost ready" for years. Every variant - Linear Transformer, Performer, Mamba, RWKV - sacrificed some quality for efficiency. KDA is the first architecture that, under controlled comparisons, simply beats full attention. If this holds up at scale, it changes the compute budget equation for every model training run.

Second, the KV cache reduction is the kind of infrastructure win that compounds. At 1 million token context, a 75% reduction in KV cache means you can serve the same throughput with roughly one-quarter the memory. For production deployments running long-context agents, RAG pipelines, or codebase analysis, that is a direct cost saving. Our guide on KV caching for transformer inference covers the mechanics of why this matters in practice.

Third, the open-source release pattern matters. Moonshot released KDA kernels and vLLM implementations alongside the paper. Combined with the K3 model weights that went live yesterday, this gives the community something rare: a path from research paper to running model that does not require a team of CUDA engineers. The break-even math on self-hosting open-weight models becomes more favorable when the architecture itself is more efficient.

Kimi K3, which we covered in depth in our developer guide and our 10-minute overview, extends KDA with Stable LatentMoE and RL improvements. It is a 2.8T MoE model with 104B activated parameters, natively multimodal, with a 1 million token context window. The fact that Moonshot scaled KDA from a 3B/48B research model to a 2.8T production model in nine months tells you how confident they are in the architecture.

None of this means KDA is the end of the attention debate. The long-context retrieval question remains open. The hybrid design still uses some full-attention layers (MLA). And competing approaches like Gated Deltanet 2, which bratao found more expressive, suggest the innovation curve is still steep. But Kimi Linear has done something the field needed: it turned a perennial "almost" into a real "yes."

Sources#

• Kimi Linear paper: arXiv:2510.26692
• Kimi K3 paper: arXiv:2607.24653
• Hacker News discussion: https://news.ycombinator.com/item?id=49082022
• Gated Deltanet 2: arXiv:2605.22791

Continue Reading#

• Kimi K3 Developer Guide: Architecture, Capabilities, and Access - deep technical walkthrough of K3, which is built on KDA
• Kimi K3 in 10 Minutes: What Developers Need to Know - quick overview of Moonshot's frontier open-weight model
• KV Caching for Transformer Inference: A Practical Guide - why KV cache efficiency matters for production deployments
• LLM Architecture Complexity: MoE, FlexAttention, and the Path Forward - broader context on where attention architectures are heading
• Frontier Model Landscape: June 2026 - how KDA-based models fit into the competitive landscape