vLLM vs TGI vs SGLang: Which Inference Server to Self-Host

Official Sources#

Last updated: August 22, 2026

Why the serving layer matters#

Picking a model is the easy part. The serving framework underneath it decides your throughput, your latency tail, how much GPU memory you waste, and how much on-call pain you sign up for. Six projects dominate self-hosted LLM inference right now: vLLM, Hugging Face TGI, SGLang, NVIDIA TensorRT-LLM, llama.cpp's server, and LMDeploy. This is not a "best one wins" post - each has a different sweet spot, and the right pick depends on your hardware, traffic shape, and how much engineering time you have.

If you're choosing a runtime for local coding-agent workloads specifically, see the companion piece: local LLM runtimes for coding agents. This post is about production-style serving of any model behind an API.

The core mechanics that separate these frameworks#

Before the comparison table, three concepts explain almost all the differences you'll see in practice:

Continuous batching. Instead of waiting for a fixed batch to finish before starting new requests, the scheduler admits and evicts requests token-by-token, keeping GPUs busy under bursty traffic. vLLM introduced this pattern broadly through its scheduler; TGI, SGLang, and TensorRT-LLM all implement variants of it now. Details: vLLM's scheduler design and Hugging Face's TGI docs.

Paged / managed KV cache. The KV cache (the memory that holds attention keys and values per generated token) is the dominant memory cost at serving time. vLLM's PagedAttention allocates the cache in non-contiguous blocks the way an OS pages virtual memory, cutting fragmentation and letting more concurrent sequences fit in the same GPU. See the original write-up: vLLM PagedAttention paper/blog. If you want the underlying transformer math first, our KV caching guide covers why this cache exists and why it's expensive in the first place.

Quantization for serving. Quantizing weights and/or the KV cache (AWQ, GPTQ, FP8, INT4/INT8) trades some accuracy for higher throughput and larger effective batch sizes by shrinking the memory footprint. Support and speedups vary a lot by framework and hardware - see vLLM's quantization docs, TensorRT-LLM's quantization toolkit, and llama.cpp's GGUF quantization formats.

Framework by framework#

vLLM#

The default choice for most teams self-hosting open-weight models on NVIDIA GPUs. PagedAttention plus continuous batching gives strong throughput on multi-request workloads, and the project ships an OpenAI-compatible server out of the box, which makes swapping in for existing OpenAI SDK clients trivial. Broad model coverage (Llama, Qwen, Mixtral, DeepSeek, and most day-one HF releases), active development, and a large community are its biggest strengths. Docs: docs.vllm.ai. Downsides: it's a big, fast-moving codebase, so version pinning matters, and peak single-request latency can trail more specialized runtimes.

Hugging Face TGI#

TGI is the "boring, it just works" option if your models already live on the Hugging Face Hub. It supports continuous batching, tensor parallelism, and quantized weights (AWQ, GPTQ, EETQ), and ships as a single Docker image with sane defaults, which lowers the ops burden versus hand-rolling a vLLM deployment. Docs: huggingface.co/docs/text-generation-inference. It has historically lagged vLLM on raw throughput benchmarks for very large concurrent loads, though the gap has narrowed release over release - check the current TGI GitHub releases for what's changed before assuming stale numbers.

SGLang#

SGLang pairs a serving runtime with its own front-end language for structured generation (constrained JSON, multi-turn agents, tool calls). Its RadixAttention scheme extends the paged-KV-cache idea to automatically share cache across requests with overlapping prefixes, which is a real win for RAG and agent workloads that repeat long system prompts. Docs: sgl-project.github.io / GitHub. It's newer than vLLM and TGI, so the ecosystem (deployment guides, Kubernetes charts, third-party integrations) is thinner, but for prompt-caching-heavy agent traffic it's worth benchmarking directly against vLLM's own prefix caching.

TensorRT-LLM#

NVIDIA's compiled-kernel approach: you build model-specific, hardware-specific engines ahead of time, and in exchange get the best raw latency and throughput on NVIDIA GPUs, especially on H100/H200/Blackwell where it exploits FP8 and newer tensor core paths. Docs: nvidia.github.io/TensorRT-LLM and NVIDIA's Triton Inference Server integration. The cost is ops complexity: engine builds are model- and GPU-specific, so adding a new model or moving to different hardware means a rebuild step, and iteration speed is slower than a Python-native server. Best fit when you're locked into a fixed model on fixed NVIDIA hardware at real scale and the engineering cost is worth the latency win.

llama.cpp server#

The lightest-weight option and the only one on this list that runs seriously well on CPU, Apple Silicon, and consumer GPUs, not just datacenter NVIDIA cards. It uses GGUF quantized formats (from Q2 up through Q8 and FP16) and exposes an OpenAI-compatible HTTP server. Docs: github.com/ggml-org/llama.cpp (see tools/server). It supports batching but its continuous-batching and multi-GPU tensor-parallel story is less sophisticated than vLLM/TGI/TensorRT-LLM, so it's the right call for single-user or small-team self-hosting and edge/local deployment, not high-concurrency multi-tenant serving. For hardware-buying guidance at this end of the spectrum, see our local LLM hardware guide.

LMDeploy#

Built by the InternLM/OpenCompass team, LMDeploy offers two backends - TurboMind (a compiled, high-throughput engine similar in spirit to TensorRT-LLM but NVIDIA-GPU-general rather than engine-per-model) and a PyTorch eager backend for broader model compatibility. It supports AWQ/GPTQ/KV-cache quantization and persistent batching. Docs: github.com/InternLM/lmdeploy. It's less widely adopted outside the InternLM ecosystem in Western deployments, so community support and third-party guides are sparser than vLLM or TGI, even though benchmarks on supported models are competitive.

Choosing by constraint, not hype#

• NVIDIA datacenter GPUs, need max throughput with broad model support and OpenAI-compatible API: start with vLLM.
• Want a single supported Docker image with minimal config, staying inside the HF ecosystem: TGI.
• Heavy agent/RAG traffic with shared long prompts, want structured-output primitives: SGLang.
• Fixed model, fixed high-end NVIDIA hardware, latency is the KPI, and you can absorb engine-build ops overhead: TensorRT-LLM.
• CPU, Apple Silicon, consumer GPU, single-user or small-team: llama.cpp server.
• Already inside the InternLM/OpenCompass tooling stack, or want a TurboMind-class engine without per-model compiles: LMDeploy.

None of these choices are permanent. All six speak (or can be fronted with) an OpenAI-compatible API, so swapping the backend later without rewriting client code is realistic - budget for a proper load test against your actual traffic pattern before committing at scale, since public benchmarks rarely match your prompt lengths, concurrency, and hardware exactly.

If you're weighing self-hosting against a managed API in the first place, our self-hosting vs. managed gateway decision guide and break-even math for self-hosting open-weights models cover that decision directly.

Continue Reading#

• Self-Hosted vs Managed AI Gateways: A Decision Guide - the cost and control tradeoffs of running your own gateway
• Ollama vs LM Studio vs vLLM vs llama.cpp: Picking a Local Runtime for Coding Agents - when the workload is local agent inference specifically
• Best AI Agent Memory Providers in 2026 - memory layers that pair with self-hosted inference

FAQ#

Which is fastest: vLLM, TGI, or SGLang?#

There is no universal answer - relative throughput depends heavily on model, hardware, batch size, and prompt-sharing patterns. All three publish their own benchmarks; run your own load test against your actual traffic before trusting a vendor number. Check the current benchmark scripts in each repo: vLLM benchmarks, TGI benchmarking tool, SGLang benchmark suite.

Do I need TensorRT-LLM for production NVIDIA deployments?#

No. TensorRT-LLM gives the best raw latency/throughput on supported NVIDIA hardware but requires per-model engine builds, which slows iteration. Many teams get to production faster and more cheaply with vLLM or TGI and only move to TensorRT-LLM once traffic and cost justify the extra ops investment.

Can llama.cpp server handle production traffic?#

It can serve real traffic, especially single-tenant or low-concurrency use cases, and it's the only option here that runs well without a datacenter GPU. For high-concurrency multi-tenant serving, vLLM, TGI, SGLang, or TensorRT-LLM have more mature continuous-batching and multi-GPU scheduling.

Do these frameworks all support quantization the same way?#

No - support and precision options vary by framework and hardware. vLLM and TGI support AWQ/GPTQ/FP8 broadly on NVIDIA GPUs, llama.cpp uses its own GGUF quantization ladder tuned for CPU/consumer GPU, and TensorRT-LLM/LMDeploy have their own quantization toolchains tied to their compiled engines. Check each project's quantization docs (linked above) for what your specific model and GPU combination supports.