Skip to main content
Watch: I Built an AI App Without Leaving Databricks

Local runtime - ggml-org - verified

llama.cpp: install, API, platforms and features

llama.cpp is a local LLM runtime from ggml-org. As its docs describe it: Developers who want a dependency-free C/C++ inference engine, with a CLI and an OpenAI-compatible server, for running GGUF models on a wide range of CPUs and GPUs locally or in the cloud. The latest release is v0.7.0, published October 15, 2026. License: MIT (open source).

Homepage - MIT

Latest release
v0.7.0
Released
2026-10-15
License
Open source
OpenAI API
Yes

Install

Copied verbatim from the page linked under each command. Labels are worded as the source words them.

macOS / Linux (curl)

Terminal
curl -LsSf https://llama.app/install.sh | sh

Source: github.com/ggml-org/llama.cpp

Windows (PowerShell)

Terminal
irm https://llama.app/install.ps1 | iex

Source: github.com/ggml-org/llama.cpp

macOS / Linux (Homebrew)

Terminal
brew install llama.cpp

Source: github.com/ggml-org/llama.cpp/blob/master/docs/install.md

Windows (Winget)

Terminal
winget install llama.cpp

Source: github.com/ggml-org/llama.cpp/blob/master/docs/install.md

Run a model

The first-run example from the official docs. OpenAI-compatible endpoint: llama-server listens on http://127.0.0.1:9931 by default (--host 127.0.0.1, --port 9931); OpenAI-compatible routes are under /v1

Terminal
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Source: github.com/ggml-org/llama.cpp

Features

What the llama.cpp docs say it supports. Not stated means the docs do not say either way.

llama.cpp documented features, each with its source
FeatureValueSource
InterfaceCLI, Server, DockerSee sources below
PlatformsmacOS, Windows, Linuxgithub.com/ggml-org/llama.cpp/blob/master/docs/install.md
BackendsCPU, CUDA, ROCm, Metal, Vulkan, SYCL, NPU, Othergithub.com/ggml-org/llama.cpp
Model formatsGGUFgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
OpenAI-compatible APIYesllama-server listens on http://127.0.0.1:9931 by default (--host 127.0.0.1, --port 9931); OpenAI-compatible routes are under /v1github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
Anthropic-compatible APIYesgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
Tool callingYesgithub.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md
Structured outputsYesgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
Vision (image input)Yesgithub.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md
Multi-GPUYesgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
MCP clientYesgithub.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
LicenseMITOpen sourcegithub.com/ggml-org/llama.cpp/blob/master/LICENSE
Latest releasev0.7.0October 15, 2026github.com/ggml-org/llama.cpp/releases/tag/v0.7.0

Models to run with llama.cpp

2 of the open-weight models in our local models directory list a command for it on their official pages:

Compare llama.cpp

FAQ

Does llama.cpp have an OpenAI-compatible API?

Yes. Endpoint, per its docs: llama-server listens on http://127.0.0.1:9931 by default (--host 127.0.0.1, --port 9931); OpenAI-compatible routes are under /v1. Source: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Does llama.cpp have an Anthropic-compatible API?

Yes, per https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Which platforms and GPUs does llama.cpp support?

Its docs list macOS, Windows and Linux, with these backends: CPU, CUDA, ROCm, Metal, Vulkan, SYCL, NPU and Other.

How do I install llama.cpp?

On macOS / Linux (curl): "curl -LsSf https://llama.app/install.sh | sh", as shown at https://github.com/ggml-org/llama.cpp. This page lists 4 install commands: macOS / Linux (curl), Windows (PowerShell), macOS / Linux (Homebrew) and Windows (Winget).

Is llama.cpp open source?

Yes. It is released under MIT; the license is at https://github.com/ggml-org/llama.cpp/blob/master/LICENSE.

What is the latest version of llama.cpp?

v0.7.0, published October 15, 2026 (https://github.com/ggml-org/llama.cpp/releases/tag/v0.7.0), as of October 18, 2026.

Which model formats does llama.cpp run?

GGUF, per https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Sources

Last verified from the pages below.

Newsletter

Local AI, every week

One email per week. New model releases, runtime updates and what they mean for what you build.

Keep exploring

Open-weight models you can run

Download sizes, licenses and the run command for each model.

All local LLM runtimes

Every runtime in one feature matrix, with sources.

Picking a runtime for coding agents

Ollama, LM Studio, vLLM and llama.cpp for agent workloads.

Setup guides

Point your coding tools at the model you picked.