Inference Internals

KV Caching: A Practical Guide to Optimizing Transformer Inference

How KV caching speeds up LLM inference - the math, the code, the memory tradeoffs, and when it stops helping. Every dev running local models hits this wall.