vLLM

vLLM is the inference server you reach for when llama.cpp is too small and cloud APIs are too expensive. Originally from UC Berkeley, vLLM pioneered PagedAttention (a KV-cache memory layout inspired by OS virtual memory paging) and continuous batching, which together give it throughput that outperforms most alternatives on the same hardware. It supports most modern model architectures, loads Hugging Face weights directly, and exposes an OpenAI-compatible HTTP endpoint. For any team self-hosting a model on an H100 or consumer GPU with serious traffic, vLLM is the production-grade default. It is less consumer-friendly than Ollama but the throughput gap is meaningful at scale.