LLM Architectures Got Complicated Fast

The original transformer architecture from "Attention Is All You Need" (2017) was elegant. Feed-forward layers, multi-head attention, residual connections. You could sketch it on a napkin. That era is over.

A recent post by Ian Barber argues that LLM architectures have crossed a complexity threshold similar to what happened with recommendation systems a decade ago. The simple stack that powered GPT-2 and early GPT-3 has been replaced by a maze of optimization techniques that are now load-bearing.

What Changed

Modern frontier models deploy a grab-bag of architectural innovations:

Attention variants everywhere. Models now use "query grouping, compressed, sparse, linear, sliding-window" attention - sometimes multiple variants in the same model. Grouped Query Attention (GQA) alone has become standard because it dramatically reduces KV cache memory during inference.

Mixture-of-Experts routing. MoE extends from feedforward blocks to the residual stream itself in recent architectures. DeepSeek-V3 routes to 256 experts per layer with only 8 active per token. The routing decision is trainable, introducing new failure modes.

Vision and audio encoders are no longer bolted on. Early multimodal approaches treated vision as a separate tower feeding into a frozen language model. Current architectures weave modality encoders throughout the stack, with cross-attention at multiple layers.

Multi-GPU comms become architectural. Once inference spans multiple GPUs, communication operations (all-reduce, all-gather) become part of the computation graph. Where you place a layer split matters for latency.

Why It Happened

Barber draws a useful parallel to recommendation systems. For most of the 2010s, recommendation models were relatively straightforward two-tower architectures with sparse embeddings. They got complicated when:

"The gap between performance being an optimization and performance being a necessity became very, very small."

At scale, every percentage point of efficiency matters. The original transformer had known inefficiencies - quadratic attention scaling, dense activations, redundant computation. One by one, researchers found optimizations. Each optimization hardened into the baseline.

The problem: once an optimization is load-bearing, you cannot easily experiment without it. Removing GQA from a modern model means you cannot fit the same context length. Removing MoE means you cannot match the same capabilities at the same inference cost. Barber puts it directly:

"You can't hand-fuse your way back without investing significant time that might not be worth it."

What HN Is Saying

The HN discussion surfaced several practical observations from people working with these models.

One commenter tracking llama.cpp development noted the implementation gap:

"The earlier models were always fully implemented. Yet with more contributors, as of today tons of latest models only have partial implementation. DeepSeekv3.2 isn't fully implemented, same with KimiK2.6, GLM5.2+, DeepSeekv4 has no implementation, MiniMaxM3 not supported yet."

The architectural diversity means inference libraries cannot keep up. Features that a model relies on may simply not exist in your preferred runtime.

Another commenter framed it as the "bitter lesson lifecycle":

"When a technique or technology is new people are making massive gains by just applying it to some use case, or gathering more data for training. As time goes on those 'bitter lesson' gains start to hit the shallow part of the logistic curve and companies have to start investing more and more effort into engineering for each small, incremental gain."

The easy scaling gains are behind us. What remains is feature engineering at the architecture level.

The Composability Problem

Barber highlights FlexAttention in PyTorch as a potential solution. FlexAttention lets you define custom attention patterns that compile down to efficient Triton kernels:

from torch.nn.attention.flex_attention import flex_attention, create_block_mask

def causal_mask(b, h, q_idx, kv_idx):
    return q_idx >= kv_idx

block_mask = create_block_mask(causal_mask, B, H, Q_LEN, KV_LEN)
output = flex_attention(query, key, value, block_mask=block_mask)

The abstraction captures sliding window, causal, document masking, and custom patterns without hand-writing CUDA. Performance hits are mild (single-digit percentage overhead vs. hand-tuned kernels).

This matters because it restores some ability to experiment. If attention is composable and fast, researchers can try new patterns without reimplementing everything from scratch.

What This Means for Developers

If you are building on top of LLMs rather than training them, three implications stand out:

Model selection is more than benchmarks. Two models with similar scores on a benchmark may have very different architectural requirements. One may need 2x the VRAM for the same context length because it uses different attention. Check the architecture, not just the numbers.

Inference libraries are fragmented. Llama.cpp, vLLM, TensorRT-LLM, and SGLang each support different model architectures to different degrees. A model's claimed features may not work in your runtime. Test with your actual stack.

Local deployment complexity is rising. Running a 7B model locally in 2023 was straightforward. Running a current MoE model with the same parameter count requires understanding expert routing, activation memory, and potentially multi-GPU splits even at small scale.

The upside: if architectural complexity is where the gains are, well-optimized inference is a genuine competitive advantage. Companies investing in inference engineering are not just saving compute costs - they are enabling capabilities that would otherwise be impractical.

The Parallel to Web Frameworks

There is an analogy in frontend development. React started simple - a render function, virtual DOM diffing, done. Then came hooks, concurrent rendering, server components, streaming, suspense boundaries. Each addition solved real problems at scale. Each addition made the mental model more complex.

LLMs are following the same arc, just faster. The transformer paper is 9 years old. The complexity explosion happened in roughly 3 years.

The question Barber poses is whether composable abstractions like FlexAttention can prevent this from becoming unmanageable. The alternative is that only large labs with custom CUDA kernels can push the frontier - a consolidation that would slow down the field.

Sources

• LLMs Are Complicated Now - Ian Barber
• HN Discussion - 120+ points, 40+ comments
• LLM Architecture Gallery - Sebastian Raschka's visual comparison tool
• FlexAttention Documentation - PyTorch