llama.cpp MoE Offload: --n-cpu-moe vs the New Expert Cache

If a mixture-of-experts model does not fit in your GPU, mainline llama.cpp now has an expert cache, --moe-cache-mib, and the right flags depend on the job. For chat and short agent turns on one large card, pair it with --cpu-moe (-cmoe): in the pull request author's RTX 5090 benchmark that generated 67.8 tokens per second against 30.8 for the default --fit layout, 2.2 times faster. For long prompts every turn, use --fit plus a cache of about 10% of the expert size, so whole expert layers stay on the GPU for prompt processing. Keep a hand-tuned --n-cpu-moe for small cards where only a sliver of the experts fits.

Last updated: October 9, 2026 - flags checked against the current llama-server README, benchmarks taken from the two merged pull requests and their review threads, including one post-merge regression report. We have not run these configurations ourselves; every number below is labelled with who measured it.

What shipped this week#

On October 7, Georgi Gerganov merged pull request #29887, "llama : add a GPU cache for MoE experts kept in host memory", and tagged it as a release highlight. The author, am17an, describes it as a port of the expert cache from the qvac-fabric fork: when experts live in system RAM, the expert matrix multiply for small batches now runs on the GPU from a least-recently-used cache of experts, and only the cache misses are uploaded over PCIe. One day later, #30112 extended it to multiple GPUs.

That matters because MoE models are the local-model story of 2026. A model like Qwen3.8-Flash-Next has 125B parameters but activates about 6B per token, so most of its weight is experts that sit idle on any given token. Until now, mainline llama.cpp gave you two ways to place them: on the GPU, or in RAM where the CPU computes them. The cache adds a third, and it is the idea third-party engines such as Strata built their speed on.

Official Sources#

Five flags that decide where experts live#

The first four flags are static: a given expert is either on the GPU or in RAM for the whole session. The cache is dynamic. Experts that keep getting picked by the router stay resident in VRAM, and an expert that has not been used recently is evicted to make room.

Two limits from the pull request decide when it helps. It only applies to batches of 32 tokens or fewer, so token generation benefits and long prompt processing falls back to the regular offload path. And it competes with whole expert layers for the same VRAM: with the cache on, --fit counts the cache and keeps fewer full expert layers on the GPU.

Benchmarks, labelled#

All of these are measured by the pull request author on Qwen3.8-Flash-Next Q4_0 (93.7 GiB, of which 65.4 GiB are experts), an EPYC 7742 with 16 threads, PCIe 4.0 x16, llama-server -fa on -c 32768 -b 2048 -ub 2048 -t 16, temperature 0, generation speed in tokens per second:

The multi-GPU pull request measured the same model on two RTX 4090s, and this is where the trade shows up. Generation went from 34.78 to 64.39 tokens per second with -cmoe --moe-cache-mib 15000, but prompt processing fell from 440.7 to 321.5 tokens per second (0.73x). The middle setting, --fit --moe-cache-mib 6544, kept 91% of prompt speed while still generating 1.74 times faster. These are the author's numbers on one model; a user's report on another model after the merge went the other way (see Where it breaks).

Independent reports from the review thread, each a single tester on their own box:

• AMD works. One tester built it with ROCm/HIP on a Radeon AI PRO R9700 (32 GB) using -cmoe to simulate a smaller card: 11.0 tokens per second with no cache, 14.5 at 8,000 MiB and 27.6 at 16,000 MiB, with decode hit rates of 85% and 94.6%. Prompt processing did not drop on that machine.
• A 16 GB card gains, by less. A fork commit that ported the pull request, linked from the review thread, reports a whole Qwen3.8-Flash-Next UD-IQ3_XXS (78 GB) on one RTX 5060 Ti 16 GB with 62 GB of RAM and --cpu-moe: generation went from 14.95 to 19.26 tokens per second (1.29x) with --moe-cache-mib 8000, and prompt speed stayed at about 89 tokens per second.
• Smaller caches gain less. Another tester on one RTX 3090 measured the cache at 1 to 5% faster than a tuned --n-cpu-moe 44 layout. A tuned manual split is a much stronger baseline than the default.
• Forcing it across two GPUs before #30112 was slower. Two testers patched out the single-GPU guard and measured 11 to 34% slower generation than their no-cache layouts, with one describing the result as PCIe-bound at a 58% hit rate (that tester did see prefill improve with the cache).

The 2.2x headline is the author's result on a 32 GB card with room for a 19,000 MiB cache; the independent single-GPU reports run from 1.01x to 2.5x depending on how much of the experts the cache can hold. The lesson is consistent: the cache pays off when it can hold a large share of the experts the router actually uses. High hit rate, big win; low hit rate, you are shipping experts over PCIe for nothing.

What it means for one agent turn#

Coding agents spend most of their wall-clock time in two phases: reading a long prompt and writing a reply. Using the author's RTX 5090 numbers, a 500-token reply takes about 16 seconds at 30.8 tokens per second and about 7 seconds at 67.8. Over a 30-turn agent session, that is roughly 8 minutes of generation down to under 4.

The prompt side moves the other way when you go all-in on the cache. Using the two-4090 numbers, reading a 20,000-token prompt takes about 45 seconds at 440.7 tokens per second and about 62 seconds at 321.5. An agent that re-reads a large context every turn can lose part of what it gained, which is why the middle --fit plus a modest cache setting exists.

Pick by task#

How to size the cache#

The pull request author's rule of thumb: about 10% of total expert size is a good starting point. For a model with 65.4 GiB of experts, that is the 6,544 MiB setting in the table above. From there:

1. Check that your build has it: llama-server --help | grep moe-cache. If nothing prints, update llama.cpp.
2. Start with --moe-cache-mib at roughly 10% of the expert size and watch the startup log. The cache prints a line with its size and the total MiB of host experts.
3. One tester read hit rates from the info-level stats line using -lv 4 and a clean exit. Another, whose cache ended up slower than no cache at a 58% decode hit rate, pointed to 75% or more, a threshold raised earlier in the review thread, as the range where it pays off. If you are well below that, the cache is too small for that model.
4. Raise the number until you run out of VRAM headroom, then compare against -cmoe with the same cache, and against your best --n-cpu-moe N without one. Keep whichever wins on your real workload, not on a short benchmark prompt.

Gerganov posted the preset he found fastest on his RTX 5090 in the review thread: cmoe = 1, moe-cache-mib = 16384, a 131,072-token context, q8_0 KV cache and 4,096 batch and micro-batch sizes, written in the INI format that llama-server's --models-preset option reads. Treat it as a starting point for a 32 GB card, not a universal answer: Gerganov reported about 1,300 tokens per second of prefill with it on a DDR5 machine, while another RTX 5090 owner with 64 GiB of DDR5-6200 reported about 500 of prefill and 40 to 60 of decode on the same config.

Where it breaks#

• Single device by design at first. The original pull request refused multiple devices. Multi-GPU arrived a day later, and the current README says the cache size is split across GPUs the same way layers are, following --tensor-split. A commenter on #30112 points out that with -ts 1,0 the second card gets no cache at all, even when it has free VRAM.
• Some models and layouts get slower, even on current master. Hours after #30112 merged, a user reported on the pull request that on their two-GPU box running GLM 5.3 Flash with the cache enabled, generation fell from about 33 to 16 tokens per second and prompt processing was about 10% slower. Their log shows a 9,991 MiB cache on the second GPU for 99,648 MiB of host experts, close to the 10% starting point, and a 43.54% hit rate on small ubatches (8 tokens or fewer). It is one report with no reply from the author yet, but it fits the pattern above: at a low hit rate the cache spends its time uploading experts over PCIe. The author's multi-GPU numbers cover one model on 2x RTX 4090, so treat other models and layouts as unproven until you benchmark them against your no-cache setup.
• Prompt processing can get slower. The cache only covers batches of 32 tokens or fewer, so moving whole expert layers off the GPU to make room for it can cut prompt speed, as the 0.73x figure above shows.
• Sampling oddities are not ruled out. One tester saw a reasoning trace degenerate into a repeated word with the cache on, and not with it off. It went away after changing top-k, so it is unconfirmed, but worth knowing if output looks wrong after enabling it.
• Not every fork follows. At least two forks with their own expert caches dropped the upstream one when merging master. If you run a fork, check which cache it actually uses.

FAQ#

What does --moe-cache-mib do in llama.cpp?#

It reserves N MiB of GPU memory as a least-recently-used cache for the MoE experts that live in system RAM. Expert computation for small batches, which is most of token generation, runs on the GPU from that cache, and only experts that are not already cached get uploaded. It is off by default (0).

Should I use --cpu-moe or --n-cpu-moe?#

--cpu-moe keeps every expert in RAM and is the cleanest pairing with a large cache: the author's best single-GPU results used it. --n-cpu-moe N keeps only the first N layers' experts in RAM, so the rest stay on the GPU as whole layers. --fit makes a similar layer-by-layer split for you, which is why the long-prompt pick above is --fit plus a modest cache rather than -cmoe. Tune --n-cpu-moe N by hand when your cache would be small: one tester on an RTX 3090 found the cache only 1 to 5% faster than a tuned --n-cpu-moe 44.

How much should --moe-cache-mib be?#

Start at about 10% of the model's total expert size, per the pull request author, then raise it while the decode hit rate climbs. On an RTX 5090 with 65.4 GiB of experts, the author's best result used 19,000 MiB. A cache near 10% is a starting point, not a guarantee: the GLM 5.3 Flash report above used about that much and ran slower than no cache.

Does the llama.cpp MoE cache work on AMD GPUs?#

A tester built and ran it with ROCm/HIP on a Radeon AI PRO R9700 and measured up to 2.5 times faster generation with a 16,000 MiB cache. A reviewer in the thread notes the implementation is backend-agnostic, but most published numbers are from NVIDIA cards.

Does it help if the model already fits in VRAM?#

No. The cache only serves experts kept in system RAM. If every expert is already on the GPU, there is nothing to cache.

Sources#

Continue Reading#

• Run Qwen3.8-Flash-Next on a Gaming PC With Strata - the third-party engine that made expert caching famous, with its own speed table for 12 GB cards
• Ollama vs LM Studio vs vLLM vs llama.cpp: Picking a Local Runtime for Coding Agents - when dropping down to raw llama.cpp flags is worth it at all
• The Best Local Coding LLMs in 2026 - which models are worth these flags in the first place
• GLM-5.2 Local Deployment: Running Z.ai's 744B Model on Consumer Hardware - MoE offloading at the extreme end, with 256 GB of RAM
• Colibri: Run GLM 5.2 on a 32GB Laptop With Disk Streaming - the same LRU-expert idea taken to disk, for machines without a big GPU