Mesh LLM: Run 235B Models Across Your Home Lab with iroh

What if you could run a 235B parameter model across your Mac Studio and a few workstations, without buying $100k worth of interconnect hardware?

Mesh LLM is a new distributed inference system that pools GPU resources across multiple machines and exposes them through a single OpenAI-compatible API. Start one node, add more later, and let the mesh figure out where to run your models. It just hit the front page of Hacker News and the discussion is worth paying attention to.

What Mesh LLM Actually Does

The core idea is simple: most developers have GPU capacity scattered across offices, closets, and workstations that sits idle most of the time. Mesh LLM lets you combine that capacity into a single inference endpoint.

The system handles three scenarios:

1. Local execution - The model fits on your machine's GPU, so it runs there
2. Routing to peers - Another node in your mesh is already running the model, so your request gets forwarded
3. Split mode ("Skippy") - The model is too large for any single machine, so it gets distributed across nodes in pipeline fashion

That third mode is where things get interesting. The Skippy engine partitions models by layer ranges - layers 0-15 on one node, 16-31 on the next, and so on down the pipeline. The inter-node communication uses iroh, a peer-to-peer networking library built on QUIC.

The iroh Foundation

Every Mesh LLM node runs an iroh endpoint that serves as its identity, public key, and only network surface. iroh handles NAT traversal and hole-punching to establish direct QUIC connections between nodes - no central server required for the data plane.

The protocol layers are well-defined:

• mesh-llm/1 - Gossip, routing, HTTP tunnels, plugin channels
• mesh-llm-control/1 - Configuration and ownership attestation
• skippy-stage/2 - Activation transport for split models

The software itself is about 18MB and presents itself as localhost:9337/v1 to any OpenAI-compatible client. Point your existing tools at it and the mesh handles the rest.

What HN is Saying

The HN discussion is largely technical, with one of the Skippy engine contributors answering questions directly.

Performance concerns are the top thread. Multiple commenters raised the obvious question: how slow is network-distributed inference compared to local RAM or even NVMe? The answer depends heavily on your setup. One benchmark mentioned in the models list shows Qwen 235B A22B running at 16 tok/s across 2 nodes - not great for interactive use, but respectable for batch work.

A detailed comment from user stymaar breaks down why this might work better than expected:

When offloading weights to RAM or NVMe, you need to transfer massive weights from slow storage to GPU for each layer being processed for each token. You're bottlenecked by DRAM bandwidth or disk read speed. When using a distributed setup, weights stay in VRAM on each machine - GPU memory bandwidth matters, not network throughput. You only transfer kilobytes of activations between stages, not gigabytes of weights.

The limiting factor is network latency. With 1ms latency and 4 nodes, you add 3ms per token - theoretical upper bound of 30 tok/s without speculative decoding. That's why this works on LANs or metro-area networks but struggles over global WAN.

Security questions came up repeatedly. The transport is encrypted via iroh's QUIC implementation, but the inference itself is not end-to-end encrypted. Nodes doing the compute can see the prompts and outputs. For medical questions to MedGemma or anything personal, you'd want a private mesh with trusted peers.

Comparisons to existing tools. Users mentioned exo, AI Horde, and cocompute.ai as alternatives. The distinction seems to be iroh's NAT traversal and the Skippy splitting engine - other tools either focus on delegation rather than splitting, or require more manual network configuration.

The Hardware Question

The demo numbers mention Mac Studios with M3 Ultra (256GB unified memory) connected via 1Gbit Ethernet, running custom Q2 quantization with sensitive tensors preserved at Q8. One contributor mentioned getting 10 tok/s on GLM 5.2 with similar hardware.

The practical threshold seems to be metro-area latency. One commenter noted that 5ms latency with jitter works fine for their home lab, but global WAN latency makes splits impractical.

The model catalog includes 40+ models from 500M parameters (laptop-friendly) to 235B MoE systems. The split mode makes the larger models accessible to hardware configurations that couldn't run them locally.

Why This Matters

The distributed inference space is heating up for a reason. Cloud API costs compound, model updates you didn't ask for break prompts, and data locality matters for an increasing number of use cases.

Mesh LLM's approach is interesting because it doesn't require exotic hardware. Most distributed training systems assume RDMA or NVLink - infrastructure that costs more than the compute itself. Using QUIC over commodity networks is a different bet: accept higher latency in exchange for zero infrastructure requirements.

The public mesh option is also worth watching. The system includes a default public mesh where anyone can contribute capacity. The incentive model isn't fully clear yet - one commenter asked about fairness guarantees and didn't get a satisfying answer - but the concept of a peer-to-peer inference network is compelling.

For individual developers or small teams, the private mesh use case is more immediately practical. Pool your own hardware, keep your data on your network, and avoid per-token billing.

Getting Started

The software is available at iroh.computer. The minimal setup is one node presenting localhost:9337/v1 to your existing OpenAI clients. Add more nodes to the mesh to expand capacity or enable split mode for larger models.

A mobile app using iroh's Swift SDK is forthcoming, along with support for ACP agent standards.

Sources

• Mesh LLM announcement post
• HN discussion with 300+ points and 70+ comments
• iroh networking library