AMD MI355X vs NVIDIA B200 vs B300 for Open-Weight Serving in 2026

Open-weights models just crossed a threshold that changes the hardware conversation. Kimi K3, the flagship open-weights release of July 2026, weighs in at 2.8 trillion parameters and needs roughly 1.5TB of VRAM at its native MXFP4 quant - before you allocate a KV cache for its 1 million token context window. A single B200 node, with 8 GPUs at 192GB each, has 1.4TB total. It does not fit.

That one fact is why this month's serving debate is not "CUDA vs ROCm" anymore. It is a capacity and price question: you serve K3 on a B300 node, you chain two B200 nodes, or you look at AMD's MI355X, the non-NVIDIA GPU that also carries 288GB of HBM per chip. Here is the comparison for teams that are about to make that choice.

Why Model Size Broke the Old Answer#

The size curve of open-weights models is accelerating. GLM5.2 shipped at 753B parameters. DeepSeek V4-Pro is 1.6T. Kimi K3 is 2.8T. Meanwhile the standard inference node - 8 GPUs with 192GB each, the B200 form factor - has not grown. At 1.4TB of HBM, a B200 node cannot hold K3's weights plus a working KV cache, which is why every deployment story for this model involves either bigger chips or two nodes.

AMD's MI355X matches the B300 on the number that matters here: 288GB of HBM per GPU. An 8-GPU MI355X node has 2.3TB, enough for the weights, a large KV pool, and headroom. That parity is the entire story - the first frontier open-weights model where HBM capacity, not raw FLOPS, decides which hardware can serve it at all.

Official Sources#

All links verified August 2, 2026.

The Three Configurations That Fit Kimi K3#

Hardware specs verified against NVIDIA's HGX platform page and AMD's Instinct product pages. GPU-hour rates are spot-market averages from gpus.io as cited by Wafer on July 31, 2026 - your actual rate depends on region and contract.

The B200 path works but pays a hidden tax: it spans two nodes, so every decode step crosses a RoCE fabric all-reduce on the critical path. The B300 node fits the model in one box and roughly doubles attention performance over B200 per NVIDIA's spec sheet. The MI355X node is the budget option with the same 288GB chips.

The Benchmark: What Wafer Measured#

Wafer published the first public serving numbers for Kimi K3 across all three configurations on July 31, 2026, using a 1,024-token input / 400-token output workload:

The headline numbers: the MI355X node delivers 3.8x the aggregate throughput of the TP16 B200 deployment and 1.3x its single-stream decode, at about 40% of the per-GPU cost. The B300 still wins raw throughput by about 1.65x over the MI355X, but costs 2.4x more per GPU, which is how the MI355X ends up at 48 tok/s/$ versus 33 for the B300.

Two caveats before anyone buys hardware off this table. First, these are self-reported numbers from Wafer, a GPU provider selling MI355X capacity - treat them as a strong signal, not an impartial lab test, and ask for a replication on your own workload. Second, the B200 number is structurally deflated: it is the only configuration that spans two nodes, and cross-node all-reduce on the decode path costs it throughput that a hypothetical single-node variant would not pay.

The Prefill Problem Is the Real AMD Gap#

Decode tok/s is only half the story. Time to first token is what users actually feel, and here the MI355X was badly behind: an identical 172k-token cold prefill took about 51 seconds on MI355X versus about 23 seconds on a B300.

The good news, and the most interesting part of Wafer's writeup, is that the gap was almost entirely one kernel. Kimi K3 on ROCm was falling back to a generic slow Triton attention path because the fast AITER MLA prefill kernel would not load - a shape mismatch where K3 at TP8 gives 12 attention heads per rank and the fast kernel only accepts 4, 8, or multiples of 16. Zero-padding the head count from 12 to 16, running the fast kernel, and extracting the real heads turned prefill from roughly 4-7k tok/s to about 13k tok/s. Not a custom kernel, not a vendor toolchain - a shape fix.

The same pattern held for speculative decoding. K3 ships no draft tensors, so Wafer paired it with RadixArk's external block-diffusion draft, hit a missing top_k_renorm_prob definition in sglang's ROCm build, and fixed it with a sort, a masked fill, and a divide in the sampling branch. That unlocked roughly 2.2x single-stream throughput and 18% more peak aggregate.

This is the real state of AMD serving in August 2026: day-0 framework support for the flagship open model, fast hardware, and a handful of small software gaps that working engineers close in an afternoon. It is no longer the multi-week kernel grind of 2025.

Decision Guide by Scenario#

Cost-constrained startup, batch and async workloads: MI355X, without much thought. At 48 tok/s/$ you can serve roughly 7x more tokens per dollar than the TP16 B200 path. Interactivity is fine at 118 tok/s single stream for most agentic workloads, and the prefill gap matters less when requests are queued rather than interactive.

Latency-sensitive, interactive product: B300. 172 tok/s single stream and 2x attention performance versus B200 make it the pick when every request is a user waiting on a first token. You pay for it - but the price gap to MI355X is exactly the premium for the TTFT guarantee.

Already standardized on NVIDIA, need K3 today: Two B200 nodes works and avoids changing your software stack, but you are paying the worst tok/s/$ of the three and the cross-node penalty. Treat it as the migration path, not the destination.

Prefill-heavy workloads - long documents, codebases, RAG over big corpora: benchmark your actual prefill on MI355X before committing. The AITER fix is in Wafer's stack but may not be in your vendor's; a 2x TTFT penalty on cold long prompts is the kind of thing that shows up in user reports, not dashboards.

The safe play either way: rent, do not buy. GPU-hour rates are falling and the Rubin generation - 288GB of HBM4 with a production ramp underway - plus AMD's MI400 series (up to 4x the theoretical MXFP4 performance of MI355X) will reset the tok/s/$ table within a year.

The Bigger Shift#

None of this makes sense without the open-weights context. K3 is the first open model with frontier-tier scores at Moonshot's API price of $3/$15 per MTok, and its weights are on Hugging Face in native MXFP4. Teams that want those economics with control over their own serving now have three viable hardware answers, and for the first time one of them is not NVIDIA. The "CUDA moat" argument is quietly becoming a per-workload cost question - and on the workload that just became the most popular open model on the internet, AMD currently wins the dollar.

Frequently Asked Questions#

Can Kimi K3 run on a B200 node?#

No. K3 needs roughly 1.5TB of VRAM for weights at native MXFP4 before any KV cache, and an 8-GPU B200 node has 1.4TB total. You need a B300 node (8x288GB), two B200 nodes, or an MI355X node (8x288GB).

How much does it cost to serve Kimi K3?#

On spot GPU rates cited by Wafer in July 2026: about $2.50 per GPU-hr on MI355X, $4.25 on B200, and $6.00 on B300. Per dollar of GPU, the MI355X delivered 48 tok/s/$ versus 7 for the TP16 B200 path and 33 for the B300.

Is AMD MI355X actually good for inference now?#

For open-weights frontier models, yes. It matches the B300 on HBM capacity (288GB per GPU), AMD shipped day-0 support for Kimi K3, and Wafer measured 952 tok/s aggregate on one node - 3.8x the two-node B200 deployment. The gaps that remain are software-level: prefill kernels and speculative-decode paths need occasional fixes on ROCm.

What is the difference between MI355X and B300?#

Both have 288GB of HBM per GPU. The B300 delivers about 1.65x the aggregate decode throughput of the MI355X (1,568 vs 952 tok/s) and roughly 2x faster cold prefill, but costs about 2.4x more per GPU. On tokens per dollar, the MI355X wins decisively at 48 vs 33 tok/s/$.

Is the CUDA moat dead?#

No - CUDA still has the maturity advantage, more frameworks with fast kernels out of the box, and the largest talent pool. But the Kimi K3 results show the moat is narrower than it was a year ago: AMD shipped day-0 support for the flagship open model and its remaining gaps were closed with small fixes, not custom kernels.

Should I buy GPUs or rent capacity for open-weights serving?#

In mid-2026, rent. Spot GPU prices are falling, the NVIDIA Rubin generation (288GB HBM4) is ramping to production, and AMD's MI400 series promises up to 4x the theoretical performance of MI355X. Committing capex today locks in a tok/s/$ table that is likely to look dated within a year.

Continue Reading#

• Kimi K3 vs K2.7: Is the Upgrade Worth It for Coding? - the model decision behind the hardware decision
• Kimi K3 Open Weights: What the Hugging Face Release Actually Changes - the MXFP4 weights and MoonEP stack, analyzed
• Where to Access Kimi K3 in 2026 - every hosting route with verified prices, from $3/$15 API to serverless
• GLM-5.2 vs DeepSeek V4 vs Qwen3: Open-Weights Coding Showdown - how the rest of the open frontier compares
• DeepSeek V4 Economics: Cost and Quality - what open-weights pricing does to the whole cost curve

Sources#

• Wafer: Is memory the moat? Running Kimi K3 at ~952 tok/s/node (July 31, 2026, verified August 2, 2026)
• NVIDIA HGX Platform specifications (verified August 2, 2026)
• AMD Instinct GPUs product page (verified August 2, 2026)
• gpus.io GPU rate tracker (rates cited by Wafer, July 31, 2026)
• RadixArk Kimi-K3-DSpark draft model (verified August 2, 2026)
• Wafer: GLM5.2 on AMD MI355X at 2626 tok/s/node (July 3, 2026)