Running Gemma 4 26B at 5 Tokens/Sec on a 13-Year-Old Xeon With No GPU

Official Sources#

A Hacker News post from Neomind Labs documents running Google's Gemma 4 26B model on a repurposed HP storage appliance from 2013. No GPU. Dual Xeon E5-2690 v2 processors (Ivy Bridge generation). Under $300 in hardware. The result: about 5.2 tokens per second decode speed - slow, but actually usable for batch work.

The interesting part isn't the benchmark numbers. It's the debugging journey that got there, and what it reveals about the state of local LLM inference on non-standard hardware.

The Hardware Setup#

The experiment used an HP StoreVirtual storage appliance, the kind of enterprise hardware that shows up on eBay after datacenter decommissions. The specs:

• Dual Xeon E5-2690 v2 processors (Ivy Bridge, circa 2013)
• DDR3 memory
• No GPU
• AVX1 support only - no AVX2, no FMA3

That last constraint is the important one. Most modern LLM inference code assumes AVX2 at minimum. Ivy Bridge predates that instruction set.

The total hardware cost was under $300. Similar setups are widely available on the used enterprise market.

The Silent Bug#

The author used ik_llama.cpp, a fork optimized for CPU inference with features like MoE routing adapted for CPU execution, speculative decoding, and CPU-ported flash attention.

The build succeeded. The model loaded. It generated text. The problem: the output was "fluent-looking multilingual gibberish."

The root cause was a silent bug in how the code handled Mixture-of-Experts operations on non-AVX2 hardware. Two graph operations - MOE_FUSED_UP_GATE and FUSED_UP_GATE - were gated on AVX2 availability at compile time but generated unconditionally at runtime. On Ivy Bridge, these operations fell through to nothing, leaving the expert FFN outputs as uninitialized memory.

The output looked reasonable because the language model structure was still working - it just had garbage where the expert computations should have been. Deterministic, NaN-free, fluent-looking nonsense.

The Fix#

The fix decomposed the fused MoE operations into separate matrix multiplication calls with fallback implementations. The fix is now upstream as PR #2138.

With the fix applied, the results on Gemma 4 26B-A4B (Q8_0 quantization):

• Decode speed: ~5.2 tokens per second
• Prompt evaluation: ~16 tokens per second

Not fast. But for batch processing, API fallback scenarios, or cost-sensitive applications, it's genuinely usable.

What HN Is Saying#

The Hacker News discussion added useful context:

The author confirmed the upstream fix. The original comment was flagged for unclear reasons, but the author reposted noting that PR #2138 contains the fix and is available for anyone with similar hardware.

Others are getting better speeds. One commenter reported 8-12 tokens/sec on a similar 13-year-old CPU, suggesting results vary significantly with context size and settings. Another shared a benchmark gist running various models on dual Xeon with 256 GB DDR4.

The "10 year old Xeon is all you need" meme continues. The thread referenced earlier discussions about running capable models on decade-old enterprise hardware.

Random aside about Android Studio timeouts. Someone asked about Android Studio disconnecting from local models after 10 minutes. No answer, but it's a reminder that tool integrations around local inference are still rough.

Why This Matters#

The conventional assumption is that local LLM inference requires either a modern GPU or recent CPUs with AVX-512. This experiment challenges that:

Aging enterprise hardware has a use case. Datacenters and enterprises discard hardware that can run 26B parameter models at usable speeds. For organizations with compliance requirements around cloud data, or developers who want inference without API costs, this is a real option.

The software isn't ready. The silent bug that produced fluent gibberish is a good example of how CPU inference paths are under-tested compared to GPU paths. If you're deploying on non-standard hardware, expect to hit edge cases.

Mixture-of-Experts models may be CPU-friendlier. Gemma 4's MoE architecture means only a subset of parameters activate per token. This makes memory bandwidth less of a bottleneck compared to dense models of similar capability.

The cost math works for some use cases. $300 in hardware plus electricity versus $0.15/million tokens from a cloud API. At low volumes, the API wins. At high volumes, or for latency-sensitive applications where you control the hardware, local inference becomes compelling.

Practical Takeaways#

If you want to replicate this:

1. Use the upstream fix. PR #2138 should be merged by now, but verify you have the MoE fallback code if targeting pre-AVX2 hardware.

2. Expect debugging. The author's experience - successful build, model loads, output looks almost right - is a common failure mode. Validate outputs against known-good implementations.

3. MoE models are the sweet spot. Dense 26B models will be slower; MoE architectures like Gemma 4-A4B only activate a fraction of weights per token.

4. Used enterprise hardware is cheap. Dual-socket Xeon systems with 128-256 GB RAM regularly sell under $500. The power draw is significant (200-300W under load) but manageable for dev/test.

The broader point: the floor for capable local inference keeps dropping. You don't need a 4090. You don't need an M-series Mac. A $300 server from 2013 will do the job, slowly but correctly - assuming you can find and fix the bugs in the software stack.

Continue Reading#

• EmbeddingGemma 2 Guide: Local Multimodal Embeddings for Code - pair a local Gemma 4 with its embedding sibling for on-device retrieval
• ACE vs ALTK-Evolve: How You Deliver Agent Memory Determines the Token Bill
• Claude Skills: A technical deep dive into Anthropic''s new approach to AI context management
• DCAS: Why Fine-Tuned Coding Agents Fall Apart When You Switch Scaffolds
• Mesh LLM: Run 235B Models Across Your Home Lab with iroh
• Microsoft PHI-4: A 14B Parameter Model That Rivals Models 5x Its Size
• Program-as-Weights Turns Prompts Into Local Fuzzy Functions