Cactus Needle 2: The 14MB Agentic LLM That Runs on a Raspberry Pi 5

Cactus Compute released Needle 2 on August 11, 2026: an open 45M-parameter model for tool calling, device use, and structured extraction that ships as a single 14MB binary and runs a complete session in 28MB of RAM. It decodes at roughly 500 tokens per second on a Raspberry Pi 5, between 400 and 1,500 tok/s on VR headsets like the Meta Quest 3S, and 300 to 700 tok/s on sub-$200 phones. The weights are Apache 2.0 on Hugging Face, the engine is a dependency-free C++ binary that also runs in WebAssembly in the browser, and the company's position is that the model trades wins with models 5x to 70x larger on tool-calling benchmarks while consuming 7x to 85x fewer FLOPs per token.

What Actually Shipped#

The whole thing is designed around one constraint: the hardware tier below PCs. Cactus points out that roughly 21 billion connected IoT devices exist against about 1.5 billion PCs, and that in emerging markets most phones ship under $200 with no NPU. Needle 2 targets budget phones, Raspberry Pis, microcontrollers, wearables, small robots, and smart home hubs.

The model is built on the Simple Attention Network architecture from the Cactus paper (arXiv 2607.18363). Instead of dense feed-forward layers, the network uses a fixed Walsh-Hadamard transform with learned diagonals, so most channel mixing costs almost no parameters. A component the authors call "engrams" moves world knowledge into hashed n-gram tables that are read a few rows per token rather than computed. Attention uses a 256-token sliding window, which caps session memory at a deterministic 28MB no matter how long the conversation runs, and the system prompt and tool declarations are pinned as permanent attention sinks so the tool definitions cannot be evicted.

The distinctive engineering decision: Needle 2 trains against its own 2-bit quantization scheme, called Cactus Quants, from pretraining through post-training - weights, activations, and KV cache alike. Conventional 2-bit post-training quantization collapses small models, so the 2-bit model you deploy is the one that was trained. The engine never decompresses weights into RAM; the 2-bit codes expand inside vector registers and fuse into integer dot products, keeping the arithmetic int8 end to end. A byte-level grammar compiled from the declared tool schemas constrains every generated token, and the engine uses it to skip up to 98% of the vocabulary projection on structural tokens.

Every response carries a learned confidence score, and off-topic requests return an empty call - the refusal - instead of a guess. Cactus frames this as edge-cloud collaboration: above a confidence threshold the device acts locally; below it, the device re-asks or escalates to a cloud model. Most device requests are routine control, so escalation stays rare.

The Benchmarks#

The headline numbers, measured end to end through the shipped C++ binary at CQ2-bit with tool retrieval on:

Scoring is ordered strict exact match: function names, call order, and every argument value must match. Cactus states the two asymmetries openly: the baselines run at f16 because 2-bit quantization collapses models not trained for it (which skews the comparison toward the baselines), and Needle is trained specifically for consumer-device tool calling while every baseline is a general language model (which skews toward Needle). Needle's gaps on BFCL concentrate exactly where its training data never went: Java, JavaScript, and the parallel multi-call categories. On Python simple calls it lands within a point of FunctionGemma, a model six times larger.

The energy argument is where the numbers get interesting. A same-shape transformer with a dense MLP spends 164 MFLOPs per token at 82M parameters; a transformer squeezed to Needle's parameter count still spends 87 MFLOPs per token because every parameter must be exercised through a matmul. Needle spends 70, with a fifth of its parameters held as gathered memory that costs no arithmetic. On device silicon, moving a byte out of flash or DRAM costs far more than a multiply-accumulate, so FLOPs per token and bytes per token together are what battery life is made of.

What Developers Are Saying#

The response in the community split into two camps. On one side, genuine enthusiasm for the form factor: the WebAssembly playground running the full model in a browser tab impressed people, and several developers said the micro-LLM class is underappreciated. The fine-tuning story - a 45M model small enough to retrain on a laptop in minutes to hours - got attention, and at least one person was already planning to compress a larger tool-calling model to 1 to 2 bits for the browser and said Needle's approach is more convenient. There were suggestions about Home Assistant plugins, hearing aids, and use as a router between small and large models, the classic hierarchy idea: small models on the device decide when to escalate to the cloud.

On the other side, the demos did not survive contact with real users. Multiple commenters pasted results where the model confidently hallucinated tool arguments: "make it a little warmer in here" produced a thermostat call to 65 degrees in cool mode, "turn on the tv" produced a lock_door call with the door set to "tv", and "5 degrees warmer" set the temperature to 5 degrees Fahrenheit. One tester asked to make a room dark and the model turned the lights on. The confidence score looked rigged in some of these failures, with 0.0158 attached to a confident-sounding wrong call. Several people noted that the marketing page reads like generated copy, and at least one found a navigation prompt a robot would fail. The honest summary from one commenter: the model is a cool idea, but humans assume more than 14MB of intelligence.

The architecture questions were the sharpest: why 2-bit instead of 4-bit with folded layers at the same size, whether the engram layers were ablated, how the confidence gate is calibrated, and whether the model can plan a DAG of tool calls where earlier results feed later parameters. No answers landed in the thread, which is the biggest open question: this design is new enough that independent verification matters.

What I Make of It#

Needle 2 is the strongest evidence yet that the tool-calling problem does not need a chat model. Turning on a light, setting a thermostat, or extracting fields from a receipt is not world knowledge; it is mapping a sentence onto typed parameters. If you strip the general chat and prose capability away, 45M parameters might genuinely be enough, and the results trading wins with 230M and 270M models at 2-bit against f16 is a real signal even with the stated skew.

The structured-output engineering is the part worth copying regardless of the model. A byte-level grammar compiled from the declared schema, enforced at every token, is a much stronger contract than JSON-mode sampling: the syntax is structurally guaranteed, and the engine converts the guarantee into a compute saving. The "empty call is the refusal" convention is also the right design for a confidence-gated system - the model has a native way to say "not mine" instead of emitting confident garbage. The failure mode seen in the thread is that calibration is hard: 0.0158 confidence on a confident-looking wrong call is a reminder that a confidence score is only useful if its calibration survives deployment, and that the escalation threshold is the actual product decision.

The realistic reading: this is not a general assistant and its authors do not claim it is. It is a device-control runtime with a 28MB ceiling that fits on parts with external RAM like the ESP32-P4, and Pebble already runs it locally in its Index 01 app. For teams building hardware products, the interesting test is not "can it chat" but "can it map 95% of real user requests to the right tool call with an empty call on the rest." If you want to try the same class of work, the LFM2.5 2.6B model covers the on-device tier for phones, and the break-even math for self-hosted open models is worth reading before you commit to a tier. For tiny devices, Needle 2 is the first model with a credible battery budget, and its release makes the cost of building a voice-controlled product with a 28MB brain something a hardware team can actually evaluate.

Continue Reading#

• LFM2.5 2.6B: The On-Device Agentic Model for Phones - the larger on-device tier, for devices with a gigabyte to spare
• Muse Glimmer 30B: Meta's Open-Weight Local Agent Model - the agent-first local model for consumer GPUs
• Turbo Fieldfare: Gemma 4 26B Running on a 2GB Mac - how far aggressive quantization can push a desktop-tier model
• Self-Hosting Open Weights Models: The Break-Even Math - when running a model yourself actually pays for itself
• VibeThinker 3B: Small Model Beats Opus on Reasoning - why the small-model class keeps outperforming its size

Sources#

• Cactus Needle 2 announcement and benchmarks
• Needle 2 weights on Hugging Face
• Needle repository on GitHub
• Simple Attention Network paper (arXiv 2607.18363)
• Show HN discussion (fetched via HN Algolia API, August 11 2026)