LLM Architectures Got Complicated Fast

Last updated: June 20, 2026

The original transformer architecture from "Attention Is All You Need" (2017) was elegant. Feed-forward layers, multi-head attention, residual connections. You could sketch it on a napkin. That era is over.

A recent post by Ian Barber argues that LLM architectures have crossed a complexity threshold similar to what happened with recommendation systems a decade ago. The simple stack that powered GPT-2 and early GPT-3 has been replaced by a maze of optimization techniques that are now load-bearing.

What Changed#

Modern frontier models deploy a grab-bag of architectural innovations:

Attention variants everywhere. Models now use "query grouping, compressed, sparse, linear, sliding-window" attention - sometimes multiple variants in the same model. Grouped Query Attention (GQA) alone has become standard because it dramatically reduces KV cache memory during inference.

Mixture-of-Experts routing. MoE extends from feedforward blocks to the residual stream itself in recent architectures. DeepSeek-V3 routes to 256 experts per layer with only 8 active per token. The routing decision is trainable, introducing new failure modes.

Vision and audio encoders are no longer bolted on. Early multimodal approaches treated vision as a separate tower feeding into a frozen language model. Current architectures weave modality encoders throughout the stack, with cross-attention at multiple layers.

Multi-GPU comms become architectural. Once inference spans multiple GPUs, communication operations (all-reduce, all-gather) become part of the computation graph. Where you place a layer split matters for latency.

Why It Happened#

Barber draws a useful parallel to recommendation systems. For most of the 2010s, recommendation models were relatively straightforward two-tower architectures with sparse embeddings. They got complicated when:

"The gap between performance being an optimization and performance being a necessity became very, very small."

At scale, every percentage point of efficiency matters. The original transformer had known inefficiencies - quadratic attention scaling, dense activations, redundant computation. One by one, researchers found optimizations. Each optimization hardened into the baseline.

The problem: once an optimization is load-bearing, you cannot easily experiment without it. Removing GQA from a modern model means you cannot fit the same context length. Removing MoE means you cannot match the same capabilities at the same inference cost. Barber puts it directly:

"You can't hand-fuse your way back without investing significant time that might not be worth it."

What HN Is Saying#

The HN discussion surfaced several practical observations from people working with these models.

One commenter tracking llama.cpp development noted the implementation gap:

"The earlier models were always fully implemented. Yet with more contributors, as of today tons of latest models only have partial implementation. DeepSeekv3.2 isn't fully implemented, same with KimiK2.6, GLM5.2+, DeepSeekv4 has no implementation, MiniMaxM3 not supported yet."

The architectural diversity means inference libraries cannot keep up. Features that a model relies on may simply not exist in your preferred runtime.

Another commenter framed it as the "bitter lesson lifecycle":

"When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training. As time goes on those 'bitter lesson' gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain."

The easy scaling gains are behind us. What remains is feature engineering at the architecture level.

The Composability Problem#

Barber highlights FlexAttention in PyTorch as a potential solution. FlexAttention lets you define custom attention patterns that compile down to efficient Triton kernels:

from torch.nn.attention.flex_attention import flex_attention, create_block_mask

def causal_mask(b, h, q_idx, kv_idx):
    return q_idx >= kv_idx

block_mask = create_block_mask(causal_mask, B, H, Q_LEN, KV_LEN)
output = flex_attention(query, key, value, block_mask=block_mask)

The abstraction captures sliding window, causal, document masking, and custom patterns without hand-writing CUDA. Performance hits are mild (single-digit percentage overhead vs. hand-tuned kernels).

This matters because it restores some ability to experiment. If attention is composable and fast, researchers can try new patterns without reimplementing everything from scratch.

What This Means for Developers#

If you are building on top of LLMs rather than training them, three implications stand out:

Model selection is more than benchmarks. Two models with similar scores on a benchmark may have very different architectural requirements. One may need 2x the VRAM for the same context length because it uses different attention. Check the architecture, not just the numbers.

Inference libraries are fragmented. Llama.cpp, vLLM, TensorRT-LLM, and SGLang each support different model architectures to different degrees. A model's claimed features may not work in your runtime. Test with your actual stack.

Local deployment complexity is rising. Running a 7B model locally in 2023 was straightforward. Running a current MoE model with the same parameter count requires understanding expert routing, activation memory, and potentially multi-GPU splits even at small scale.

The upside: if architectural complexity is where the gains are, well-optimized inference is a genuine competitive advantage. Companies investing in inference engineering are not just saving compute costs - they are enabling capabilities that would otherwise be impractical.

The Parallel to Web Frameworks#

There is an analogy in frontend development. React started simple - a render function, virtual DOM diffing, done. Then came hooks, concurrent rendering, server components, streaming, suspense boundaries. Each addition solved real problems at scale. Each addition made the mental model more complex.

LLMs are following the same arc, just faster. The transformer paper is 9 years old. The complexity explosion happened in roughly 3 years.

The question Barber poses is whether composable abstractions like FlexAttention can prevent this from becoming unmanageable. The alternative is that only large labs with custom CUDA kernels can push the frontier - a consolidation that would slow down the field.

FAQ#

Why are LLM architectures so complicated now?#

At scale, every percentage point of efficiency matters. The original transformer had known inefficiencies like quadratic attention scaling and dense activations, and each optimization that fixed one hardened into the baseline. Ian Barber compares it to recommendation systems, which got complicated once performance went from an optimization to a necessity.

How does Mixture-of-Experts work in models like DeepSeek-V3?#

MoE routes each token to a subset of expert networks instead of running every parameter. DeepSeek-V3 routes to 256 experts per layer with only 8 active per token, and the routing decision is trainable, which introduces new failure modes.

What is FlexAttention in PyTorch?#

FlexAttention lets you define custom attention patterns, such as causal, sliding window or document masking, that compile down to efficient Triton kernels without hand-writing CUDA. The post reports single-digit percentage overhead versus hand-tuned kernels, which restores some ability to experiment with new attention patterns.

Why doesn't llama.cpp support every new model?#

The architectural diversity means inference libraries cannot keep up. A Hacker News commenter tracking llama.cpp noted that many recent models are only partially implemented or not supported at all, so features a model relies on may not exist in your preferred runtime. The post's advice is to test with your actual stack.

Continue Reading#

• Detecting LLM Text with Classical ML: TF-IDF Still Works
• The Claude Tokenizer Change: What ~30% More Tokens Means for Your Bill
• DeepSeek Pauses Fundraising After Leaked Investor Transcript Reveals Compute Gap
• Ilya Sutskever's 30 Papers: The Reading List That Covers 90% of What Matters
• Transformers.js: Run AI Models Directly in the Browser

Sources#

• LLMs Are Complicated Now - Ian Barber
• HN Discussion - 120+ points, 40+ comments
• LLM Architecture Gallery - Sebastian Raschka's visual comparison tool
• FlexAttention Documentation - PyTorch