The best terminal assistant has no AI

TERMy answers “what was that tar flag again” in under two milliseconds, offline, with zero tokens billed and zero chance of inventing a flag that doesn’t exist. It is a terminal assistant built with no LLM anywhere in the path, and that omission is the entire design, not a limitation someone is apologizing for. It handles the most common terminal friction - forgotten flags, command recall, syntax you knew last month - with deterministic lookups against things that are already correct on your machine. The output is faster, cheaper, and more reliable than any model-backed equivalent, and it stays that way under load.

The interesting part is not the tool. It is what the tool’s existence proves. A large share of what gets marketed as “AI-assisted terminal work” is deterministic retrieval wearing a probabilistic costume. When the input space is bounded and the right answer already lives in a man page, a help output, or your own shell history, a model is not adding intelligence. It is adding a network hop, a per-call cost, a sampling step, and a non-zero probability of a confident wrong answer to a problem that a sorted index solved decades ago.

So treat TERMy less as a product and more as a boundary marker. It draws the line where traditional automation still beats AI-assisted workflows outright: bounded inputs, known-correct answers, latency that has to feel instant, offline requirements, and high call frequency. On the near side of that line, deterministic systems win on every axis that matters in production. Cross it with an LLM and you pay more money for a weaker guarantee. The value of a project like this is that it makes the line visible instead of leaving teams to discover it after they have shipped a slow, expensive helper for a fast, cheap problem.

Underneath, TERMy is a retrieval system, not a reasoning system. It builds a local index over the things that already describe your environment - installed commands, parsed man pages and help text, and your shell history - and serves queries through a fuzzy matcher and a set of rules. A query comes in, it hits the index, it ranks candidates by relevance and recency, and it returns a result. There is no tokenization of intent, no inference, no sampling. The same input produces the same output every time, and it does so without leaving the machine. That determinism is not a nice-to-have; it is the property that makes the tool trustworthy at the speed people actually type.

Compare that to the flow inside an LLM-backed terminal assistant. Natural language gets tokenized, shipped over the network, run through model inference, sampled into a response, and often round-tripped again if the model decides to call a tool. Every one of those stages adds latency measured in hundreds of milliseconds to seconds, a cost that scales with call volume, and a small but permanent chance of producing an answer that reads perfectly and is simply false. For open-ended reasoning that trade is often worth it. For “which flag lists files by size,” you have taken a lookup and rebuilt it as a generation problem, inheriting all of generation’s failure modes and none of its upside.

The mechanism worth internalizing is that the deciding variable is the shape of the problem, not the sophistication of the model. When correct answers are enumerable and already indexed, retrieval dominates generation on latency, cost, determinism, and offline capability at the same time. A model’s real strength is generalizing over inputs nobody enumerated in advance, and that strength is wasted - sometimes actively harmful - the moment the input space is closed. TERMy performs well not because it is cleverer than a model, but because it is matched to a problem where cleverness was never the constraint. Precision, speed, and repeatability were.

Where people go wrong is treating “AI-assisted” as a strict upgrade over what came before. The reflex now is to reach for an LLM by default, because typing a question in plain English feels better than remembering a query syntax, and that feeling gets mistaken for capability. It is a category error. The comfort of the interface is real, but it says nothing about whether generation is the right engine underneath it. Defaulting to the most powerful available tool, rather than the one that fits the problem, is how teams end up paying model prices to answer questions a static table already knew.

The concrete anti-pattern is wrapping a deterministic lookup inside a probabilistic model and calling it progress. A team builds a “smart CLI helper” that fires an API call to answer how to list files by size, or how to kill a process on a port, or what the flag for recursive copy is. Each of those has one correct, stable answer that a rule returns in microseconds. By routing it through a model, they have introduced a network dependency, a recurring cost per keystroke-level interaction, latency the user feels on every query, and the possibility of a hallucinated flag that fails silently or, worse, does something destructive. Complexity went up, reliability went down, and the user experience got slower in exchange for a friendlier prompt box.

What TERMy actually surfaces is that this is an architectural decision, not an ideological one. The question is narrow and answerable: is the input space bounded, and does a correct answer already exist somewhere retrievable. If yes, deterministic automation is the right call and a model is overhead. If the input is genuinely open-ended and no precomputed answer exists - explaining an unfamiliar codebase’s build system, composing a novel multi-stage command from a vague description - that is where a model earns its cost and its latency. Most teams never draw that line explicitly, so they apply one engine to both sides of it. The teams that do draw it end up using LLMs less often and getting more out of them when they do, because they have stopped spending model budget on lookup-table problems.

The working pattern is a router sitting in front of two engines, not a single model trying to be everything. A query comes in, a cheap classifier decides whether the input is bounded or open-ended, and it routes accordingly. Bounded queries - the ones with a known-correct answer already sitting in a man page, a help output, or shell history - go to a deterministic retrieval path. Open-ended queries, where no precomputed answer exists, go to generation. The classifier does not need to be smart. Pattern matching on query structure, a short allow-list of known intents, and a confidence threshold covers most of it. The point is not a perfect gate; it is defaulting to retrieval and escalating to generation only when retrieval has nothing to return.

The deterministic side is an index and a matcher, and both are boring by design. On install and on a refresh trigger, you walk the PATH, parse the man pages and —help output for every command, and fold in shell history with timestamps. That produces a flat, sorted structure you can query in microseconds. A fuzzy matcher handles the gap between what the user typed and what the entry is actually called, and ranking leans on two signals: relevance to the query and recency of use, because the flag you reached for last week is more likely the one you want than one you have never touched. Refresh is the only moving part - you re-index when packages change, not on every query, so the cost of freshness is paid once, off the hot path.

The part most teams skip is that the same index makes generation safe on the rare occasions you need it. When a query escalates to the model - composing a novel multi-stage command, say - you do not ship the model’s output straight to the user’s shell. You parse the command it produced and check every flag and subcommand against the index you already built. A flag the model invented does not exist in the parsed man page, so it gets caught before it ever runs. That turns the deterministic system into a verifier for the probabilistic one: the model proposes, the index disposes. You get generation’s flexibility on the open set without inheriting its habit of confidently emitting flags that were never real.

Instrument the split from day one. Log what fraction of queries resolve on the deterministic path versus the generative one, and track latency and cost per path. In most terminal-assistant workloads that ratio lands heavily on the retrieval side - the forgotten flag, the command recall, the syntax you knew last month - which means you have removed the network hop and the per-call charge from the overwhelming majority of interactions. That number is the whole argument. If ninety percent of queries never touch a model, you are paying model prices on ten percent of traffic and serving the rest at the speed of a local lookup.

A platform team ships an internal CLI helper backed by an LLM. It works in the demo. Every query - how do I list files by size, kill the process on port 8080, recursive copy with permissions - fires an API call, waits on inference, and returns an answer that is usually right. Usage climbs, and so does the bill. Latency sits between one and three seconds per query because every interaction is a round trip. Then the helper suggests a flag that does not exist on the installed version of a tool, a developer pastes it, and the command fails in a way that costs an afternoon to trace.

The team pulls the usage logs and finds the shape of the problem. The large majority of queries are lookups with one stable answer - flag recall, command syntax, process-and-port questions - and only a thin slice are genuinely open-ended. So they put a router in front. The lookups now resolve against a local index of parsed man pages and history in under a few milliseconds, offline, at zero marginal cost. The API bill drops in proportion to the traffic that used to hit the model, and the median query goes from seconds to instant. Nothing about the model changed; the routing did.

The residual open-ended queries still go to the model, and this is where the earlier bill was actually buying something - turning a vague description into a multi-stage pipeline no static table could hold. But now the model’s output passes through the index before it reaches the shell, so the invented flag from before gets flagged as nonexistent instead of executed. The team ends up using the LLM on a fraction of the traffic, spending less, shipping faster responses, and getting a stronger correctness guarantee on the model’s output than they had when it ran unchecked on everything.

Match the engine to the shape of the problem, not to the sophistication you happen to have available. When the input space is bounded and the correct answer already exists somewhere retrievable, deterministic retrieval wins on latency, cost, determinism, and offline capability at the same time - there is no axis on which generation catches up. When the input is open-ended and no precomputed answer exists, generation earns its cost. The mistake is not using LLMs. The mistake is using one engine for both sides of a line you never drew.

The architectural default should be deterministic-first, with generation as an escalation, not the reverse. Most teams build generation-first because the plain-English interface feels like the product, then bolt caching and guardrails on afterward to claw back the cost and reliability they gave away. Flip it. Start with the index, route to the model only when retrieval returns nothing, and use the index to verify whatever the model produces. You will reach for the LLM less often and get more out of it when you do, because you have stopped spending its budget on problems a sorted table solved decades ago.

TERMy’s real output is not a faster way to remember tar flags. It is a visible line between the two engines, drawn in a place most teams only find after they have shipped a slow, expensive helper for a fast, cheap problem. The skill that matters now is knowing which side of that line a given problem sits on before you build for it. Reaching for the most powerful tool by default is not engineering; it is deferring the decision and paying for the deferral on every call. Draw the line first. Then most of what looked like it needed a model turns out to need an index, and the model gets to do the one thing it is genuinely better at.