INFERENCE

Magnitude launched as an Apache-2.0 inference engine that compiles and tunes kernels on your own hardware, claims up to 2x faster decode than llama.cpp, and wires itself into OpenCode, Codex and Claude Code. Here is what shipped, what early testers measured, and where it breaks.

Cloudflare published the serving playbook behind Workers AI running Moonshot Kimi K2.6 and Zhipu GLM 5.2: FP8 KV caches double Kimi's resident context to 1.37M tokens, INT4 weights shrink GLM 5.2's checkpoint 40%, and a page-tagging integrity check protects the shared cache at under 1% overhead. The numbers show what actually matters when open frontier models run on GPU fleets.

AMD MI355X vs NVIDIA B200 vs B300 for serving Kimi K3's about 1.5TB of open weights: verified specs, the Wafer benchmark, and cost per token for each.

A new multi-model orchestration system routes requests across open-weight models to match frontier performance at reduced inference cost. Here is what we know.

A fair comparison of vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp, and LMDeploy for self-hosted LLM inference - batching, quantization, hardware, and ops.

Google Trends put CBRS stock on the board after Cerebras' first public-company earnings. The developer takeaway is not a trade. It is that AI inference demand is now being priced, questioned, and audited in public.

Google released DiffusionGemma today, a 26B MoE open model that generates entire 256-token blocks in parallel instead of one token at a time. Here is what that means for latency, local inference, and the post-autoregressive landscape.

How KV caching speeds up LLM inference - the math, the code, the memory tradeoffs, and when it stops helping. Every dev running local models hits this wall.

C++ inference engine for LLMs. GGUF format, quantization, CPU and Metal/CUDA support. The foundation most local tools build on.

High-throughput inference server for LLMs. PagedAttention memory management. The go-to for serious local or self-hosted serving.

Apple's array framework for machine learning on Apple Silicon. Native Metal support, unified memory, first-class LLM inference.

Run 50,000+ ML models with a simple API. No infrastructure management. Pay-per-second billing. Deploy custom models with Cog. Popular for image generation and audio.

Fastest inference for open-source models. 200+ models via unified API. Ranks #1 on speed benchmarks for DeepSeek, Qwen, Kimi, and Llama. Serverless pay-per-token pricing.

LPU-powered inference delivering 500-1,000+ tokens/sec. Purpose-built chip with on-chip SRAM instead of HBM. 5-10x faster than GPU providers. Free tier available.

Wafer-scale AI inference at 3,000+ tokens/sec. The WSE-3 chip has 4 trillion transistors and 900K AI cores. 20x faster than GPU providers. OpenAI partnership for inference.