llama.cpp

llama.cpp is the C++ inference library from Georgi Gerganov that most of the local-LLM ecosystem is built on. Ollama, LM Studio, Jan, and GPT4All all ship a wrapped or forked llama.cpp at their core. Using it directly gives you the lowest overhead and the most control: quantization down to 2-bit, Metal acceleration on Apple Silicon, CUDA on NVIDIA, Vulkan on AMD, and first-class support for the GGUF weight format that the entire local ecosystem has standardized on. For developers who want to embed local inference in a product rather than wrap an existing tool, llama.cpp is the layer to integrate against. The bundled llama-server binary exposes an OpenAI-compatible HTTP endpoint so any existing tool can swap to a local model with one config change.