Colibri: Running GLM 5.2 on a 32GB Laptop with Disk Streaming and Expert Offloading

A developer with a 12-core laptop and 32GB of RAM got GLM 5.2 running locally. Not a quantized 7B parameter model - the full 744B Mixture-of-Experts flagship. The project, Colibri, hit the Hacker News front page with 730+ points and 180 comments. The HN thread reflects equal parts admiration for the hacker spirit and practical questions about when this approach makes sense.

The Core Insight

GLM 5.2's MoE architecture activates only ~40B parameters per token out of its 744B total. Of those, only ~11GB changes from token to token (the routed experts). The rest - attention layers, shared experts, embeddings (~17B parameters) - stays constant.

Colibri exploits this by:

1. Keeping the dense part resident in RAM at int4 quantization (~9.9GB)
2. Storing routed experts on disk (~370GB at int4, ~19MB per expert)
3. Streaming experts on demand with per-layer LRU caching and OS page cache as a free L2

The engine is a single C file - c/glm.c at ~1,300 lines. No BLAS, no Python at runtime, no GPU required.

Performance Numbers

The author is upfront: this is slow. Initial reports mention 0.1 tokens per second. But that was never the point. From the HN submission:

"The important thing was the journey to reach this goal. I just wanted it to work at all costs, even slowly."

Community benchmarks show better results on faster hardware. Users with NVMe drives and more RAM report usable speeds, though still far below cloud API performance.

What HN Is Saying

The hacker spirit resonates:

The top comment simply states: "This is the hacker spirit." The author replied: "Thank you so much, it's true! It all started with this spirit!"

This energy pervades the thread. Multiple commenters compared Colibri to antirez's ds4 project (the creator of Redis working on a similar disk-streaming approach for GLM 5.2). The author confirmed inspiration: "Antirez is the number one!"

SSD wear concerns:

Several asked about disk lifespan. The README addresses this directly with an SSD wear warning. The author clarified that heavy writes are limited to the KV cache, while expert reads dominate. Architectures with unified memory (like Apple Silicon) can keep the KV cache in RAM entirely.

Practical alternatives:

A pragmatic commenter noted: "For most projects the more practical solution is to use clouds offering GLM 5.2 for free. 1 token per minute is minuscule compared to their rate limits for free usage."

This is true. For production use, cloud APIs are faster and cheaper. But that misses what Colibri demonstrates: that the architectural constraints of MoE models enable approaches previously thought impossible.

Agent integration:

Users asked whether Colibri can plug into coding agents like Claude Code or Pi. The author confirmed work is underway: "We're working on it right now with a pull request that will also arrive for opencode!"

This would enable fully local agentic workflows, though at reduced speed.

Hardware pricing concerns:

A subthread lamented current RAM and SSD prices. One commenter's shopping cart went "from $399 to $475" for basic DDR5. Another observed that affordable local inference is getting harder as hardware costs rise. Yet others pointed out that used or budget hardware can hit the necessary specs for under $600.

Technical Details

The architecture breaks down like this:

The 21,504 routed experts come from 75 MoE layers with 256 experts each, plus the MTP head. At runtime, only a small subset is active per token, and the LRU cache keeps hot experts in memory.

No external dependencies means the project compiles anywhere with a C compiler. The tradeoff is reimplementing functionality that libraries would provide, but for a research/hobby project, the simplicity has value.

Related Work

Colibri isn't the only project exploring disk-based inference:

• antirez's ds4: The Redis creator has a GLM 5.2 branch using similar SSD streaming techniques. Reports suggest usable speeds on a 128GB M5 MacBook Pro.
• llama.cpp: The standard for local inference, though it typically expects models to fit in memory or uses mmap for slower streaming.
• ExLlamaV2: GPU-focused but exploring similar expert-level offloading strategies.

The common thread is exploiting MoE sparsity. When only ~5% of parameters are active per token, you don't need the entire model in fast memory.

When This Makes Sense

Colibri fills a specific niche:

1. Learning and experimentation: Understanding how MoE inference actually works at the systems level
2. Fully offline operation: No network dependency, no API costs, no data leaving your machine
3. Proof of concept: Demonstrating that consumer hardware can run frontier models

It does not make sense for:

• Production workloads requiring speed
• Cost optimization (cloud APIs are cheaper per token)
• General coding assistance (too slow for interactive use)

The author is clear-eyed about this: "I don't have that hardware so I can't test it on hardware that is more powerful than my computer."

What's Next

The project is actively developed. The author is working on:

• OpenCode integration for agentic workflows
• Performance improvements to reduce streaming overhead
• Community contributions (the README welcomes participation)

For developers interested in low-level LLM inference, Colibri offers a readable codebase. At 1,300 lines of C, you can understand the entire system in an afternoon. That's rare for ML inference code.

The project embodies a principle that resonates with HN's audience: software doesn't have to be practical to be valuable. Sometimes you build something just to prove it can be done.

Sources

• Colibri GitHub repository
• Hacker News discussion
• antirez ds4 GLM 5.2 branch
• GLM 5.2 architecture documentation