Topic
INFERENCE
All blog posts, tools, and guides about Inference from Developers Digest.
13 resources - 6 posts, 7 tools
Blog Posts
View in blog →
Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net
Cloudflare published the serving playbook behind Workers AI running Moonshot Kimi K2.6 and Zhipu GLM 5.2: FP8 KV caches double Kimi's resident context to 1.37M tokens, INT4 weights shrink GLM 5.2's checkpoint 40%, and a page-tagging integrity check protects the shared cache at under 1% overhead. The numbers show what actually matters when open frontier models run on GPU fleets.
7 min read
AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving in 2026
Kimi K3 open weights need roughly 1.5TB of VRAM, which does not fit on a B200 node. That forces a real hardware decision: B300, two B200 nodes, or AMD's MI355X. Here is the head-to-head with verified specs, the Wafer benchmark, and what it costs per token.
9 min read
Echo Claims Fable-Level Results at One-Third the Cost Using Open-Weight Models
A new multi-model orchestration system routes requests across open-weight models to match frontier performance at reduced inference cost. Here is what we know.
7 min read
Cerebras Stock Is a Public Test of AI Inference Demand
Google Trends put CBRS stock on the board after Cerebras' first public-company earnings. The developer takeaway is not a trade. It is that AI inference demand is now being priced, questioned, and audited in public.
7 min read
DiffusionGemma: Google Bets Diffusion Can Make Text Generation 4x Faster
Google released DiffusionGemma today, a 26B MoE open model that generates entire 256-token blocks in parallel instead of one token at a time. Here is what that means for latency, local inference, and the post-autoregressive landscape.
8 min read
KV Caching: A Practical Guide to Optimizing Transformer Inference
How KV caching speeds up LLM inference - the math, the code, the memory tradeoffs, and when it stops helping. Every dev running local models hits this wall.
11 min read
Related Tools
All tools →Replicate
Run 50,000+ ML models with a simple API. No infrastructure management. Pay-per-second billing. Deploy custom models with Cog. Popular for image generation and audio.
InfrastructureTogether AI
Fastest inference for open-source models. 200+ models via unified API. Ranks #1 on speed benchmarks for DeepSeek, Qwen, Kimi, and Llama. Serverless pay-per-token pricing.
InfrastructureGroq
LPU-powered inference delivering 500-1,000+ tokens/sec. Purpose-built chip with on-chip SRAM instead of HBM. 5-10x faster than GPU providers. Free tier available.
InfrastructureCerebras
Wafer-scale AI inference at 3,000+ tokens/sec. The WSE-3 chip has 4 trillion transistors and 900K AI cores. 20x faster than GPU providers. OpenAI partnership for inference.
Infrastructurellama.cpp
C++ inference engine for LLMs. GGUF format, quantization, CPU and Metal/CUDA support. The foundation most local tools build on.
Local AIvLLM
High-throughput inference server for LLMs. PagedAttention memory management. The go-to for serious local or self-hosted serving.
Local AIMLX
Apple's array framework for machine learning on Apple Silicon. Native Metal support, unified memory, first-class LLM inference.
Local AIKeep exploring
More on Inference
- - Replicate - recommended Inference tool from the Developers Digest directory
- - Compare Tools - dive deeper across the Developers Digest knowledge base
- - All Inference articles in the blog archive
- - Developers Digest on YouTube - video tutorials covering Inference and more
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever
Explore 826 topics
Browse All Topics