Most automation doesn't need a smart model

Needle 3 ships as a single binary between 8 and 29 MB, small enough to run on a watch, a robot vacuum or a microcontroller with nothing behind it. Cactus built it for three narrow jobs done offline: read the tools an app exposes and emit the right calls with their arguments filled, pull typed fields out of unstructured text, and return a sentence embedding for local search and routing. The claim is that it beats models ten times its size on mobile tool calls and matches models two to three times bigger on extraction. What it trades away to get there is general chat.

Underneath it is a 29 to 121M-parameter Simple Attention Network quantised to two bits, in a format Cactus calls CQ2, trained on 360B tokens of a proprietary structured dataset. On a Raspberry Pi 5 it decodes at 400 to 4,000 tokens per second and prefills at 1,000 to 10,000.

The structural idea is intelligence laddering. Every layer is a sub-network with more capacity than the one below it, so you pick a depth from 2 layers to 20 to match the device and the task. The small end is the interesting part: a 2-layer subnetwork fine-tuned on one product’s tool set runs on hardware the full model would never fit. Cactus reports that tuning on DroidCall lifts every subnetwork by 18 to 36 points, and from 4 layers up the tuned subnetwork passes DeepSeek V4 Flash, starting at 29M parameters. Constraining the model to one well-defined task is what buys the accuracy.

In code it is a decorated function.

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])

The signature gives the argument types, the docstring is the tool description, and run() closes the loop: the model picks the call, Needle runs your function, feeds the result back, and hands you the final response. Because the model reads the schema literally, the description is the whole game. A narrow tool with a plain description beats a broad one: one tool per action, enum options named after what a user actually says, value formats spelled out in the description (“City, ST”), and a small toolset per turn, since every extra tool is a chance to misroute.

Two mechanisms matter if you are putting this anywhere it can act on the world.

The first is triggers, regexes matched against the request. A match restricts decoding to the matched tools and forces a call, even below the confidence floor. That helps when a description cannot enumerate every phrasing and you need a request to reach a specific tool. It also means a triggered tool always fires on a match, so a catch-all pattern has to explicitly exclude the nouns other tools own, or “switch the fan on and dim the kitchen lights” ends up at the wrong one.

The second is the confidence score, and this is where triggers bite. Every response carries a calibrated confidence, and the engine already applies a floor of 0.1: below it, the call is withheld into suppressed_calls and function_calls comes back empty. Above the floor the score is yours. The documented pattern is to act when it is high, show the call and ask when it is middling, and treat an empty result as a refusal.

r = agent.complete(user_text)
calls = r["function_calls"]
held = r["suppressed_calls"]
if calls and r["confidence"] >= 0.7:
    execute(calls) # sure: act
elif calls or held:
    confirm(calls or held, r["reasoning"]) # unsure: show the call, ask
else:
    say("I can't do that here") # nothing to do: refuse

Because a triggered tool produces a call regardless of confidence, the score is the only thing left telling you whether to run it or confirm it. A triggered tool wired to a consequential action with no score check will fire on every match, which is rarely what you want for something that spends money or unlocks a door.

Fine-tuning is LoRA on the frozen base at the full 20 layers, then a 4-bit .cact export of whatever subnetwork you ship; the Python package does both. Deployment is a prebuilt engine under 1 MB per target that loads the weights at start, so needle build --platform linux-arm64 --layers 8 gives you an 8-layer subnetwork on a Pi with nothing to compile on the device.