Cactus Needle 3: 8–29MB on-device models that punch above their weight at tool calls

Cactus Compute has released Needle 3, a foundation model aimed squarely at edge hardware — phones, wearables, robots, smart-home devices, cars, and even microcontrollers. The entire model ships as a single 8–29 MB binary built on the company’s Simple Attention Network, and it makes a deliberate trade: it sacrifices general conversational ability to specialize in three narrow tasks. On mobile tool calls it claims to beat models ten times its size, and on structured extraction it says it matches models two to three times larger. The three capabilities are function calling (picking the right app-exposed tools and filling their arguments, returning an empty list rather than guessing when nothing fits), schema-driven extraction with a decode grammar that guarantees parseable JSON output, and on-device text embeddings for local search and routing. Everything runs offline, with no cloud or hub round trip.

The headline technical idea is ‘intelligence laddering’: the network is structured so each layer is a sub-network of monotonically increasing capacity, letting developers pick anything from a 2-layer to a 20-layer variant. Cactus says the 4-layer subnetwork can match DeepSeek V4 Flash after a single epoch of task-specific fine-tuning. The models range from 29–121M parameters using aggressive CQ2 2-bit quantization, were trained on 360B tokens of proprietary structured data, and reportedly hit 400–4,000 tokens/s decode on a Raspberry Pi 5. Developers integrate via a Python package that pulls the inference engine from Hugging Face; tools are declared by decorating functions, with optional regex ‘triggers’ to force routing and a calibrated confidence score for deciding whether to act, confirm, or refuse.

The significance is less about raw intelligence than about where inference can now live. By collapsing tool calling and extraction into a few-megabyte binary that runs locally at usable speeds, Needle 3 targets the class of assistant features that don’t need a large language model at all — turning natural-language requests into deterministic, typed actions on-device. That keeps user data off the network, removes cloud latency and cost, and makes the approach viable on hardware far below what hosted LLMs require. Source is available on GitHub.