Inference Tutorials, Tools, and Guides

New7 min read

DeepInfer: What DeepSeek Open-Sourced and What You Can Run

DeepSeek open-sourced DeepInfer, the inference engine behind its API. It is offline-only, V4.1-Flash-only and Blackwell-or-Ascend-only. What it is, what is missing, and who benefits.

DeepSeek Open-Weight Models Inference

6 min read

Magnitude: A Self-Tuning Inference Engine Bids to Be Your Agent's Local Backend

Magnitude is an Apache-2.0 inference engine that tunes kernels on your hardware and plugs into OpenCode, Codex and Claude Code. What shipped, where it breaks.

News Local LLMs Open Source

7 min read

Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net

Cloudflare runs Kimi K2.6 and GLM 5.2 on Workers AI with FP8 KV caches, INT4 weights and a KV cache integrity check. What each technique buys on GPU fleets.

News Cloudflare Workers AI

9 min read

AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving

AMD MI355X vs NVIDIA B200 vs B300 for serving Kimi K3's about 1.5TB of open weights: verified specs, the Wafer benchmark, and cost per token for each.

AMD NVIDIA GPU

7 min read

Echo Claims Fable-Level Results at One-Third the Cost Using Open-Weight Models

A new multi-model orchestration system routes requests across open-weight models to match frontier performance at reduced inference cost. Here is what we know.

News Hacker News AI Models

7 min read

vLLM vs TGI vs SGLang: Which Inference Server to Self-Host

A fair comparison of vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp, and LMDeploy for self-hosted LLM inference - batching, quantization, hardware, and ops.

Inference vLLM Self-Hosting

7 min read

Cerebras Stock Is a Public Test of AI Inference Demand

CBRS stock trended after Cerebras' first public earnings. What public-market scrutiny of fast AI inference means for developers - not investment advice.

AI Infrastructure Cerebras Markets

8 min read

DiffusionGemma: Google Bets Diffusion Can Make Text Generation 4x Faster

DiffusionGemma is Google's 26B MoE open model that generates 256-token blocks in parallel. What diffusion generation means for latency and local inference.

ai open-source inference

11 min read

KV Caching: A Practical Guide to Optimizing Transformer Inference

How KV caching speeds up LLM inference - the math, the code, the memory tradeoffs, and when it stops helping. Every dev running local models hits this wall.

LLM Inference Optimization

New7 min read

DeepInfer: What DeepSeek Open-Sourced and What You Can Run

DeepSeek open-sourced DeepInfer, the inference engine behind its API. It is offline-only, V4.1-Flash-only and Blackwell-or-Ascend-only. What it is, what is missing, and who benefits.

DeepSeek Open-Weight Models Inference

6 min read

Magnitude: A Self-Tuning Inference Engine Bids to Be Your Agent's Local Backend

Magnitude is an Apache-2.0 inference engine that tunes kernels on your hardware and plugs into OpenCode, Codex and Claude Code. What shipped, where it breaks.

News Local LLMs Open Source

7 min read

Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net

Cloudflare runs Kimi K2.6 and GLM 5.2 on Workers AI with FP8 KV caches, INT4 weights and a KV cache integrity check. What each technique buys on GPU fleets.

News Cloudflare Workers AI

9 min read

AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving

AMD MI355X vs NVIDIA B200 vs B300 for serving Kimi K3's about 1.5TB of open weights: verified specs, the Wafer benchmark, and cost per token for each.

AMD NVIDIA GPU

7 min read

Echo Claims Fable-Level Results at One-Third the Cost Using Open-Weight Models

A new multi-model orchestration system routes requests across open-weight models to match frontier performance at reduced inference cost. Here is what we know.

News Hacker News AI Models

7 min read

vLLM vs TGI vs SGLang: Which Inference Server to Self-Host

A fair comparison of vLLM, TGI, SGLang, TensorRT-LLM, llama.cpp, and LMDeploy for self-hosted LLM inference - batching, quantization, hardware, and ops.

Inference vLLM Self-Hosting

7 min read

Cerebras Stock Is a Public Test of AI Inference Demand

CBRS stock trended after Cerebras' first public earnings. What public-market scrutiny of fast AI inference means for developers - not investment advice.

AI Infrastructure Cerebras Markets

8 min read

DiffusionGemma: Google Bets Diffusion Can Make Text Generation 4x Faster

DiffusionGemma is Google's 26B MoE open model that generates 256-token blocks in parallel. What diffusion generation means for latency and local inference.

ai open-source inference

11 min read

KV Caching: A Practical Guide to Optimizing Transformer Inference

How KV caching speeds up LLM inference - the math, the code, the memory tradeoffs, and when it stops helping. Every dev running local models hits this wall.

LLM Inference Optimization

INFERENCE

GLM-5.3 on Workers AI Leaked Other Users' Prompts: SGLang Fix List

DeepInfer: What DeepSeek Open-Sourced and What You Can Run

Magnitude: A Self-Tuning Inference Engine Bids to Be Your Agent's Local Backend

Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net

AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving

Echo Claims Fable-Level Results at One-Third the Cost Using Open-Weight Models

vLLM vs TGI vs SGLang: Which Inference Server to Self-Host

Cerebras Stock Is a Public Test of AI Inference Demand

DiffusionGemma: Google Bets Diffusion Can Make Text Generation 4x Faster

KV Caching: A Practical Guide to Optimizing Transformer Inference

Keep exploring Inference

INFERENCE

GLM-5.3 on Workers AI Leaked Other Users' Prompts: SGLang Fix List

DeepInfer: What DeepSeek Open-Sourced and What You Can Run

Magnitude: A Self-Tuning Inference Engine Bids to Be Your Agent's Local Backend

Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net

AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving

Echo Claims Fable-Level Results at One-Third the Cost Using Open-Weight Models

vLLM vs TGI vs SGLang: Which Inference Server to Self-Host

Cerebras Stock Is a Public Test of AI Inference Demand

DiffusionGemma: Google Bets Diffusion Can Make Text Generation 4x Faster

KV Caching: A Practical Guide to Optimizing Transformer Inference

Keep exploring Inference

INFERENCE

GLM-5.3 on Workers AI Leaked Other Users' Prompts: SGLang Fix List

Keep exploring Inference

Get Smarter About AI Dev

INFERENCE

GLM-5.3 on Workers AI Leaked Other Users' Prompts: SGLang Fix List

Keep exploring Inference

Get Smarter About AI Dev