Cloudflare Runs Kimi and GLM at Scale: FP8 KV Caches, INT4 Weights, and a Cache Safety Net

On August 3, as part of Agents Week, Cloudflare published the serving playbook behind Workers AI running two of the most demanding open models it hosts: Moonshot's Kimi K2.6 and Zhipu's GLM 5.2. Both are large, long-context mixture-of-experts models, and both are memory-bound on the GPU. The post details three techniques layered on top of Cloudflare's existing large-model serving work: quantizing the KV cache, compressing model weights, and protecting the shared cache those two optimizations expose. The measured numbers make it one of the most concrete serving write-ups of the year.

What shipped#

Cloudflare serves these models on GPUs in its own data centers via Workers AI, using SGLang as the inference engine. The company says SGLang offers the best performance in the market and that it upstreams patches to the project. Three optimizations make up the new work:

1. FP8 KV caches. As a model generates, it stores attention keys and values for every processed token in the KV cache, and for a long-context model the cache fills GPU memory before the weights do. Cloudflare stores the cache in FP8 (e4m3) instead of BF16, halving its size. On Kimi K2.6 that raises resident context from roughly 686,000 tokens to about 1.37 million, twice as much.

The benefit is not raw speed. On a disaggregated H200 deployment, BF16 is a few percent faster per token at any single concurrency level, because the FP8 attention kernel converts values as it reads them. But BF16 runs out of cache at 32 concurrent requests and cannot admit a 33rd, while FP8 keeps going to 64 and reaches 2,192 tokens per second, about 41% higher than BF16's peak, for roughly 30% less cost per token. Because prefill and decode run as separate pools, prefill keeps the BF16 cache: prefill is compute-bound rather than memory-bound, and BF16's slightly higher throughput wins there. Across Cloudflare's evaluation suite, FP8 and BF16 caches are indistinguishable in accuracy.

2. INT4 weights. For GLM 5.2, Cloudflare compresses weights from FP8 to INT4. The checkpoint drops from 705 GB to 421 GB, about 40%, and per-GPU memory on an 8-way tensor-parallel deployment falls from roughly 88 GB to 52 GB, leaving room for about 1.18 million tokens of KV cache on the same hardware.

Decode gets faster because generating each token means streaming weights out of GPU memory, and decode speed is limited by memory bandwidth: move less data and every token arrives sooner. The effect is largest at low concurrency, where per-request latency matters most. Prefill behaves the opposite way, because INT4 weights must be expanded before matrix multiply: GLM sustains about 10,160 tokens per second of prefill in FP8 versus 8,660 in INT4. The disaggregated design turns this into a choice rather than a compromise, so Cloudflare runs INT4 for decode and FP8 for prefill. Accuracy stays within 0.8 points of the FP8 model across every benchmark it runs.

3. KV cache integrity checking. Both optimizations pack hundreds of requests onto one GPU, all reading and writing pages of the same physical KV cache. Paged attention, continuous batching, and cache reuse rely on exact bookkeeping, and at Cloudflare's request volumes, even a one-in-a-billion mistake shows up regularly. The fix: every physical cache page gets a tag that changes whenever the page is reallocated, and the server records which pages and tags each request expects. Before supported decode operations read from the cache, the mappings are checked, and a mismatch aborts the request rather than returning data from the wrong page. The validation runs as a separate batch check instead of being fused into the attention kernel, keeping the cost under 1% on both throughput and tail latency. It is opt-in per deployment, and the default path uses a no-op tracker with no measurable overhead.

Why it matters to developers#

Three takeaways stand out.

First, this is the real economics of open frontier models. The GPU costs of serving Kimi K2.6 or GLM 5.2 are dominated by memory, not compute. Halving the cache and cutting the weights 40% do not change answers, they change how many customers fit on a GPU. A 30% cut in cost per token on a model this large is the difference between viable and not viable for hosted open-weights inference. For anyone who has run the numbers on self-hosting open-weights models, this is the operator-side version of that math, and the direction is the same: quantized serving is table stakes, not an optimization.

Second, disaggregation converts tradeoffs into choices. The classic dilemma with quantization is that it helps one phase and hurts the other. Cloudflare sidesteps it by running INT4 decode pools and FP8 prefill pools, and the FP8 KV cache lands in decode only. This is the same architecture pattern that makes KV caching interesting in the first place: when memory is the bottleneck, where you place precision is a routing decision.

Third, the integrity check is the quiet interesting part. Multitenant GPU serving is a shared-memory system, and page-reuse bugs in paged attention are a known class of correctness risk. A tagging scheme that aborts mismatched reads, measured at under 1% overhead, is a defensible answer to the question of whether sharing hardware cheaply is safe. Cloudflare's plan to make it a fleet-wide default is worth watching.

On the practical side, the work being upstreamed to SGLang means the techniques are not locked behind Workers AI. If you self-host, the vLLM vs TGI vs SGLang comparison is the right starting point, and FP8 KV caches and quantized weights are increasingly supported across all three engines. For the models themselves, the Kimi K3 open-weights release and the GLM 5.2 cost math frame why anyone cares about serving these two families cheaply at all.

The direction of travel is clear: expect FP8 KV caches to expand across Cloudflare's fleet, NVFP4 weight validation on Blackwell, and integrity checks left on everywhere at negligible cost. For developers, hosted open frontier models keep getting cheaper per token, with the same answers.

Continue Reading#

• vLLM vs TGI vs SGLang: Which Inference Server to Self-Host
• KV Caching: A Practical Guide to Optimizing Transformer Inference
• Kimi K3 Weights Land on HuggingFace: 2.8T Open Frontier Model You Can Actually Download
• GLM-5.2 Cost Math: When Open-Weights Coding Models Actually Save You Money
• Self-Hosting Open-Weights Models: The Real Break-Even Math

Sources#

• Smaller, faster, safer: running Kimi and GLM at scale - Cloudflare Blog
• Serving large models on Workers AI - Cloudflare Blog
• Welcome to Agents Week - Cloudflare Blog
• SGLang: Fast Serving Framework for Large Language Models and Vision Language Models - GitHub