Needle 2: a 14MB, 45M-parameter model that runs tool-calling on a microcontroller

Cactus Compute has released Needle 2, an Apache-2.0 model aimed squarely at cheap edge hardware rather than the Macs and PCs that dominate on-device AI. At 45M parameters, it ships as a single dependency-free 14MB C++ binary that fits a full session into roughly 28MB of RAM — small enough to run on an ESP32-S3 microcontroller, a Raspberry Pi 5 (~500 tokens/sec decode), sub-$200 Android phones, and VR headsets. The company frames the target market bluntly: with over 21 billion IoT devices against ~1.5 billion PCs, most of the edge is low-cost gear with no GPU, no NPU, and a few hundred MB of memory. Pebble already runs it locally in its Index 01 app to turn speech into actions offline.

The core bet is that tool calling, device control, and structured extraction don’t need a frontier model’s world knowledge — the hard part is mapping a messy sentence onto typed functions, so 45M parameters can trade benchmark wins with rivals like FunctionGemma 270M and LFM2.5 230M while being 5–70x smaller. Every response is a call envelope constrained by a byte-level grammar compiled from the declared schema, an empty call signals refusal, and a learned confidence score decides whether to act locally or escalate to the cloud — keeping the default path private, instant, and free. The same schema-as-interface approach doubles as classification and list extraction from documents.

The efficiency comes from architectural and inference choices co-designed for the target silicon. Needle is trained against 2-bit Cactus Quants from pretraining onward — weights, activations, and KV cache — so the deployed model is the trained model, with no post-hoc quantization loss. A Hadamard MLP, hashed n-gram “engram” tables for world knowledge, multi-lane residual streams, and a 256-token sliding window (with tool declarations pinned so they can’t be evicted) cut both FLOPs and, more importantly, bytes read per token — the real energy cost on flash-constrained devices. The engine never decompresses weights into RAM and skips up to 98% of the vocabulary projection on grammar-forced tokens. Weights are on Hugging Face, and the 45M size means the model can be fine-tuned on a laptop to speak a specific product’s tool vocabulary.