Local runtimes - verified
Local LLM runtimes compared
12 ways to run an open model on your own hardware: which platforms and GPUs each one supports, which model formats it loads, whether it serves an OpenAI- or Anthropic-compatible API, and the install command, copied from official docs. Looking for the models themselves? See open-weight models you can run locally.
- Runtimes
- 12
- Open source
- 10
- OpenAI API
- 12
- Anthropic API
- 9
Feature matrix
Yes and No link to the doc that says so. Not stated means the docs do not say either way. Scroll the table sideways on a small screen.
| Runtime | Platforms | Backends | Formats | OpenAI API | Anthropic API | Tool calling | Structured outputs | Vision | Multi-GPU | MCP client | License | Latest release |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ollama | macOS, Windows, Linux | CPU, CUDA, ROCm, Metal, Vulkan | GGUF, Safetensors | Yes | Yes | Yes | Yes | Yes | Yes | Not stated | Open sourceMIT | v0.40.6 |
| LM Studio | macOS, Windows, Linux | CPU, CUDA, Vulkan, ROCm, Metal | GGUF, MLX | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Not open sourceLM Studio Terms of Use for the app, MIT for the lms CLI | 0.4.26 |
| llama.cpp | macOS, Windows, Linux | CPU, CUDA, ROCm, Metal, Vulkan, SYCL, NPU, Other | GGUF | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Open sourceMIT | v0.7.0 |
| vLLM | Linux, macOS | CUDA, ROCm, CPU, TPU, Metal, NPU, Other | Safetensors, GGUF | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Open sourceApache-2.0 | v0.31.0 |
| SGLang | Linux, macOS | CUDA, ROCm, CPU, TPU, NPU, Metal, Other | Safetensors, GGUF, MLX | Yes | Yes | Yes | Yes | Yes | Yes | Not stated | Open sourceApache-2.0 | v0.5.21 |
| MLX-LM | macOS, Linux | Metal, CUDA, CPU | MLX | Yes | Not stated | Not stated | Not stated | Not stated | Not stated | Not stated | Open sourceMIT | v0.32.2 |
| Jan | macOS, Windows, Linux | CPU, CUDA, Vulkan, Metal | GGUF, MLX | Yes | Not stated | Yes | Yes | Yes | Yes | Yes | Open sourceApache-2.0 | v0.8.6 |
| LocalAI | macOS, Linux, Windows | CPU, CUDA, ROCm, SYCL, Metal, Vulkan | GGUF, Safetensors, MLX | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Open sourceMIT | v4.11.0 |
| Docker Model Runner | macOS, Windows, Linux | CPU, CUDA, ROCm, Metal, Vulkan, Other | GGUF, Safetensors, Other | Yes | Yes | Yes | Yes | Yes | Yes | Not stated | Open sourceApache-2.0 | v1.2.8 |
| KoboldCpp | Windows, Linux, macOS | CPU, CUDA, Vulkan, Metal, ROCm | GGUF, Other | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Open sourceAGPL-3.0 | v1.123.1 |
| Lemonade | Windows, Linux, macOS | CPU, CUDA, ROCm, Vulkan, Metal, NPU | GGUF, ONNX, Other | Yes | Yes | Yes | Not stated | Yes | Not stated | Not stated | Open sourceApache-2.0 | v2026.42.1 |
| Microsoft Foundry Local | Windows, macOS, Linux | CPU, CUDA, NPU, Other | ONNX | Yes | Not stated | Yes | Not stated | Not stated | Not stated | Not stated | Not open sourceMicrosoft Software License Terms for the CLI, MIT for the SDK | v2.1.0 |
Runtime by runtime
Who each runtime is for, as its own docs describe it. Each page has the install commands, the feature table with a source for every value, and the comparisons it appears in.
Ollama
v0.40.6Ollama
Developers who want to download and run open models from the command line, with a local REST API on port 11434 and built-in launchers for coding agents such as Claude Code and Codex.
LM Studio
0.4.26Element Labs, Inc.
People who want a desktop app with a chat interface for downloading and running GGUF and MLX models locally, plus a local OpenAI-compatible server and a headless daemon (llmster) for servers and CI.
llama.cpp
v0.7.0ggml-org
Developers who want a dependency-free C/C++ inference engine, with a CLI and an OpenAI-compatible server, for running GGUF models on a wide range of CPUs and GPUs locally or in the cloud.
vLLM
v0.31.0vLLM project
A high-throughput, memory-efficient inference and serving engine for LLMs, used to serve models over an OpenAI-compatible HTTP server or run offline batch inference in Python, mainly on Linux GPU and accelerator hardware.
SGLang
v0.5.21SGLang project (LMSYS)
A high-performance serving framework for large language and multimodal models, designed for low-latency, high-throughput inference from a single GPU up to large distributed clusters.
MLX-LM
v0.32.2Apple (ml-explore)
A Python package for generating text and fine-tuning large language models on Apple silicon with MLX, offering a CLI, a Python API and a basic local HTTP server.
Jan
v0.8.6Menlo Research (janhq)
Jan is an open-source desktop app for downloading and running local models offline on macOS, Windows and Linux, with an optional OpenAI-compatible local API server.
LocalAI
v4.11.0LocalAI team (Ettore Di Giacinto)
LocalAI is an open-source, self-hosted engine that serves LLMs and other model types through OpenAI- and Anthropic-compatible APIs on CPU-only or GPU hardware.
Docker Model Runner
v1.2.8Docker
Docker Model Runner is for developers who want to pull, run and serve AI models with Docker tooling from Docker Hub, any OCI-compliant registry or Hugging Face.
KoboldCpp
v1.123.1LostRuins (Concedo) / KoboldAI community
People who want to run GGUF models locally from a single executable with no installation, using a bundled KoboldAI Lite web UI for chat, story writing and roleplay plus a local API for other apps.
Lemonade
v2026.42.1lemonade-sdk (community project sponsored by AMD)
People who want a local AI server that runs LLMs and other models on their own CPU, GPU or NPU and exposes them through OpenAI, Anthropic and Ollama compatible APIs, with optimizations for AMD Ryzen AI, Radeon and Strix Halo PCs.
Microsoft Foundry Local
v2.1.0Microsoft
Developers who want to ship on-device AI inside their own applications using native SDKs, a curated catalog of optimized ONNX models and automatic NPU, GPU or CPU acceleration for a single user per device.
Compare two runtimes
Side by side, field by field, with the differences marked and both install commands.
Method
How to read this table
Read from each runtime's official docs, README and release pages on . Runtimes ship weekly; check the linked source before you rely on a feature.
- Yes, No, Not stated
- Yes and No appear only when the runtime's own docs say so, and each links to the page that says it. Not stated means the docs neither claim nor rule out the feature; it is not a No.
- Commands copied verbatim
- Install and run commands come from the official page linked under each one, unchanged. Platform labels are worded as those pages word them.
- Not a benchmark
- Nothing here measures speed, quality or memory use. The table lists documented capabilities only; the order follows the data file, not a score.
FAQ
Which local runtimes expose an OpenAI-compatible API?
All 12: Ollama, LM Studio, llama.cpp, vLLM, SGLang, MLX-LM, Jan, LocalAI, Docker Model Runner, KoboldCpp, Lemonade and Microsoft Foundry Local. Each runtime's page gives the default endpoint its docs list, so a tool that speaks the OpenAI API can point at it.
Which local runtimes expose an Anthropic-compatible API?
9 of 12 document an Anthropic-compatible (Messages API) endpoint: Ollama, LM Studio, llama.cpp, vLLM, SGLang, LocalAI, Docker Model Runner, KoboldCpp and Lemonade. The docs of MLX-LM, Jan and Microsoft Foundry Local do not say.
Which local LLM runtimes are open source?
10 of 12 ship under open-source licenses: Ollama (MIT), llama.cpp (MIT), vLLM (Apache-2.0), SGLang (Apache-2.0), MLX-LM (MIT), Jan (Apache-2.0), LocalAI (MIT), Docker Model Runner (Apache-2.0), KoboldCpp (AGPL-3.0), Lemonade (Apache-2.0). LM Studio is not open source (LM Studio Terms of Use for the app, MIT for the lms CLI); Microsoft Foundry Local is not open source (Microsoft Software License Terms for the CLI, MIT for the SDK).
Which local runtimes run on Windows, macOS and Linux?
By the platforms their docs list, all 12 list Linux, all 12 list macOS, and 9 list Windows (Ollama, LM Studio, llama.cpp, Jan, LocalAI, Docker Model Runner, KoboldCpp, Lemonade and Microsoft Foundry Local). vLLM, SGLang and MLX-LM do not list Windows.
Which local runtimes use the GPU on a Mac?
11 of 12 list Metal as a backend: Ollama, LM Studio, llama.cpp, vLLM, SGLang, MLX-LM, Jan, LocalAI, Docker Model Runner, KoboldCpp and Lemonade. 5 also run models in Apple's MLX format: LM Studio, SGLang, MLX-LM, Jan and LocalAI.
Which local runtimes can split a model across multiple GPUs?
9 of 12 document multi-GPU support: Ollama, LM Studio, llama.cpp, vLLM, SGLang, Jan, LocalAI, Docker Model Runner and KoboldCpp. The docs of MLX-LM, Lemonade and Microsoft Foundry Local do not say.
Which local runtimes can connect to MCP servers?
6 of 12 document acting as an MCP client: LM Studio, llama.cpp, vLLM, Jan, LocalAI and KoboldCpp. The docs of Ollama, SGLang, MLX-LM, Docker Model Runner, Lemonade and Microsoft Foundry Local do not say.
What does "Not stated" mean in the table?
That the runtime's official docs, read on October 18, 2026, neither claim nor rule out the feature. It is not a No: the feature may exist without being documented. Yes and No are shown only when the docs say so, and each one links to the page that says it.
Is this a benchmark or a ranking?
No. The table lists what each runtime's docs say it supports and how to install it. It measures neither speed nor quality, and the order follows the data file, not a score.
Newsletter
Local AI, every week
One email per week. New model releases, runtime updates and what they mean for what you build.