PagedAttention

In depth

An attention algorithm that stores the KV cache in fixed-size blocks that do not need to be contiguous in GPU memory, borrowing the virtual memory and paging idea from operating systems. Allocating cache on demand nearly eliminates the fragmentation and over-reservation that limited batch sizes in earlier servers, and lets requests share blocks for common prefixes or parallel samples. It is the core of vLLM, whose paper reported 2-4x higher throughput than FasterTransformer and Orca at the same latency.

Primary source: arxiv.org