Turning a decision model into a chatbot, one next-symbol vote at a time

jevchat is a for-fun experiment that coerces a classification/decision API (referred to as “Jev”) into generating text, even though it was never meant to produce free-form output. Rather than asking the model to write, the tool asks it to choose: given the question and the reply written so far, which symbol comes next? The model returns a probability for each option in some alphabet plus a STOP token, a sampler draws from that normalized distribution, appends the result, and repeats until STOP is chosen. It’s a clever inversion — building a generator out of a thing that only knows how to rank finished choices.

Most of the project is the machinery around that core trick. Alphabets range from lowercase letters and raw ASCII to whole-word and BPE token lists of up to several thousand entries, and sampling strategies trade cost against quality: a single “choice” question over the whole alphabet, a “bisect” that narrows via earlier/later yes-no splits, “buckets” that shard large vocabularies across many questions, and a “refine” pass that rescores the winners. Beam search, temperature, top-p/top-k, ensembling of reshuffled prompts, and a repetition penalty are all exposed as flags. The most consequential design choice is “presentation”: asking the model to rank the fully-appended candidate strings (hypothesis mode) rather than the bare symbols roughly triples top-1 accuracy on character alphabets, because ranking completed strings is exactly what a decision model is built to do.

The author is candid that the results are a lousy, impractical chatbot — every emitted symbol costs an API call, so latency and price are brutal — and frames the whole thing as a joke that happens to illustrate how classification and generation relate. It was largely built with Claude implementing the sampling algorithms from the author’s descriptions. The repo ships a live terminal readout of generation rate and the model’s top-scoring symbols, plus 158 offline tests using a fake client and mocked HTTP transport, so everything but the actual benchmark runs without a key or network.