QUANTIZATION

How much VRAM a local LLM needs: Q4 weights are about 30% of BF16, plus a KV cache that grows per token. Real numbers from 8 GB to 256 GB.

PrismML's Bonsai 27B uses 1-bit quantization to compress a 27B model to 3.9GB - small enough to run on an iPhone. Here's how it works and what HN thinks.

Run GLM-5.2 locally with Unsloth dynamic quantization: hardware math for a 256GB Mac or 24GB GPU, quantization tradeoffs and lessons from the HN thread.

C++ inference engine for LLMs. GGUF format, quantization, CPU and Metal/CUDA support. The foundation most local tools build on.