Local AI

vLLM

High-throughput inference server for LLMs. PagedAttention memory management. The go-to for serious local or self-hosted serving.

Try vLLMgithub.com/vllm-project/vllm

Save

vLLM is the inference server you reach for when llama.cpp is too small and cloud APIs are too expensive. Originally from UC Berkeley, vLLM pioneered PagedAttention (a KV-cache memory layout inspired by OS virtual memory paging) and continuous batching, which together give it throughput that outperforms most alternatives on the same hardware. It supports most modern model architectures, loads Hugging Face weights directly, and exposes an OpenAI-compatible HTTP endpoint. For any team self-hosting a model on an H100 or consumer GPU with serious traffic, vLLM is the production-grade default. It is less consumer-friendly than Ollama but the throughput gap is meaningful at scale.

inference server throughput self-hosted gpu pagedattention

vLLM at a glance

Last verified October 17, 2026

Pricing: vLLM - free and open source (Apache-2.0)
Install: uv pip install vllm --torch-backend=auto
vllm download-kernels
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
Models: Serves Hugging Face (or ModelScope) models by name: vllm serve <model>, for example vllm serve Qwen/Qwen2.5-1.5B-Instruct, starts an OpenAI-compatible server. Covers decoder-only, MoE, hybrid state-space, multimodal, embedding and reward models on NVIDIA, AMD and Intel GPUs, TPUs, CPUs, Ascend NPUs and Apple Silicon.
Recent changes: Oct 5v0.31.0 speeds up DeepSeek-V4.1-Flash with FlashMLA mega attention, adds fast restarts with vllm preload, and removes tokenizer_mode="slow".
Sep 22v0.30.0 adds DeepSeek-V4.1-Flash and GLM-5.3-Flash and a persistent per-GPU weight-cache daemon for fast starts; scale-out endpoints now need --enable-scale-out.
Sep 9v0.29.0 makes Model Runner V2 the default for all models, adds Hy4-preview and Qwen3.8-Flash-Next, and removes ten deprecated architectures.

Sources: vllm-project/vllm README, vLLM quickstart, GitHub releases, v0.25.0 release notes

Similar Tools

Local AI

Ollama

The easiest way to run LLMs locally. One command to pull and run any model. OpenAI-compatible API. 52M+ monthly downloads. Supports GGUF, Safetensors, and custom Modelfiles.

Local AI

LocalAI

Open-source OpenAI API replacement. Runs LLMs, vision, voice, image, and video models on any hardware, no GPU required. 60+ backends. Distributed mode for scaling.

Local AI

llama.cpp

C++ inference engine for LLMs. GGUF format, quantization, CPU and Metal/CUDA support. The foundation most local tools build on.

Local AI

MLX

Apple's array framework for machine learning on Apple Silicon. Native Metal support, unified memory, first-class LLM inference.

Get started with vLLM

High-throughput inference server for LLMs. PagedAttention memory management. The go-to for serious local or self-hosted serving.

Try vLLM

Get weekly tool reviews

Honest takes on AI dev tools, frameworks, and infrastructure - delivered to your inbox.

Subscribe Free

Compare all pricing Compare side by side vLLM install, API and features

More Local AI Tools

Ollama

The easiest way to run LLMs locally. One command to pull and run any model. OpenAI-compatible API. 52M+ monthly downloads. Supports GGUF, Safetensors, and custom Modelfiles.

LM Studio

Desktop app for discovering, downloading, and running local LLMs. Clean chat UI, OpenAI-compatible API server, and automatic GPU detection. MLX engine optimized for Apple Silicon.

Jan

Open-source ChatGPT alternative that runs models fully offline. Desktop app with local models, cloud API connections, custom assistants, and MCP integration. Apache 2.0 licensed.