Skip to main content
Watch: I Built an AI App Without Leaving Databricks

Local runtime - vLLM project - verified

vLLM: install, API, platforms and features

vLLM is a local LLM runtime from vLLM project. As its docs describe it: A high-throughput, memory-efficient inference and serving engine for LLMs, used to serve models over an OpenAI-compatible HTTP server or run offline batch inference in Python, mainly on Linux GPU and accelerator hardware. The latest release is v0.31.0, published October 5, 2026. License: Apache-2.0 (open source).

Homepage - Source on GitHub - Apache-2.0

Latest release
v0.31.0
Released
2026-10-05
License
Open source
OpenAI API
Yes

Install

Copied verbatim from the page linked under each command. Labels are worded as the source words them.

linux (NVIDIA CUDA)

Terminal
uv pip install vllm --torch-backend=auto

Source: docs.vllm.ai/en/latest/getting_started/quickstart

linux (AMD ROCm)

Terminal
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

Source: docs.vllm.ai/en/latest/getting_started/quickstart

Google TPU

Terminal
uv pip install vllm-tpu

Source: docs.vllm.ai/en/latest/getting_started/quickstart

docker (CUDA 13.0)

Terminal
docker pull vllm/vllm-openai:v0.31.0

Source: github.com/vllm-project/vllm/releases/tag/v0.31.0

Run a model

The first-run example from the official docs. OpenAI-compatible endpoint: http://localhost:8000/v1 (server starts at http://localhost:8000 by default; change with --host and --port)

Terminal
vllm serve Qwen/Qwen2.5-1.5B-Instruct

Source: docs.vllm.ai/en/latest/getting_started/quickstart

Features

What the vLLM docs say it supports. Not stated means the docs do not say either way.

vLLM documented features, each with its source
FeatureValueSource
InterfaceCLI, Server, Python library, DockerSee sources below
PlatformsLinux, macOSdocs.vllm.ai/en/latest/getting_started/installation/gpu
BackendsCUDA, ROCm, CPU, TPU, Metal, NPU, Otherdocs.vllm.ai/en/latest/getting_started/quickstart
Model formatsSafetensors, GGUFdocs.vllm.ai/en/latest/features/quantization/gguf
OpenAI-compatible APIYeshttp://localhost:8000/v1 (server starts at http://localhost:8000 by default; change with --host and --port)docs.vllm.ai/en/latest/getting_started/quickstart
Anthropic-compatible APIYesdocs.vllm.ai/en/latest/serving/online_serving
Tool callingYesdocs.vllm.ai/en/latest/features/tool_calling
Structured outputsYesdocs.vllm.ai/en/latest/features/structured_outputs
Vision (image input)Yesdocs.vllm.ai/en/latest/features/multimodal_inputs
Multi-GPUYesdocs.vllm.ai/en/latest/serving/parallelism_scaling
MCP clientYesdocs.vllm.ai/en/stable/examples/tool_calling/openai_responses_client_with_mcp_tools
LicenseApache-2.0Open sourcegithub.com/vllm-project/vllm/blob/main/LICENSE
Latest releasev0.31.0October 5, 2026github.com/vllm-project/vllm/releases/tag/v0.31.0

Models to run with vLLM

31 of the open-weight models in our local models directory list a command for it on their official pages:

Compare vLLM

FAQ

Does vLLM have an OpenAI-compatible API?

Yes. Endpoint, per its docs: http://localhost:8000/v1 (server starts at http://localhost:8000 by default; change with --host and --port). Source: https://docs.vllm.ai/en/latest/getting_started/quickstart/

Does vLLM have an Anthropic-compatible API?

Yes, per https://docs.vllm.ai/en/latest/serving/online_serving/

Which platforms and GPUs does vLLM support?

Its docs list Linux and macOS, with these backends: CUDA, ROCm, CPU, TPU, Metal, NPU and Other.

How do I install vLLM?

On linux (NVIDIA CUDA): "uv pip install vllm --torch-backend=auto", as shown at https://docs.vllm.ai/en/latest/getting_started/quickstart/. This page lists 4 install commands: linux (NVIDIA CUDA), linux (AMD ROCm), Google TPU and docker (CUDA 13.0).

Is vLLM open source?

Yes. It is released under Apache-2.0; the license is at https://github.com/vllm-project/vllm/blob/main/LICENSE.

What is the latest version of vLLM?

v0.31.0, published October 5, 2026 (https://github.com/vllm-project/vllm/releases/tag/v0.31.0), as of October 18, 2026.

Which model formats does vLLM run?

Safetensors and GGUF, per https://docs.vllm.ai/en/latest/features/quantization/gguf/

Sources

Last verified from the pages below.

Newsletter

Local AI, every week

One email per week. New model releases, runtime updates and what they mean for what you build.

Keep exploring

Open-weight models you can run

Download sizes, licenses and the run command for each model.

All local LLM runtimes

Every runtime in one feature matrix, with sources.

Picking a runtime for coding agents

Ollama, LM Studio, vLLM and llama.cpp for agent workloads.

Setup guides

Point your coding tools at the model you picked.