Let cheap LLMs hallucinate labels, then map them back with embeddings

Constraining an LLM to a fixed taxonomy—say, sorting an e-commerce query like “wood coffee table” into one of several hundred Wayfair categories—usually means shipping the entire list of legal values to the model as a structured-output schema (a giant Pydantic Literal). It works, but it’s expensive, has to be resent on every call, and eventually bumps against input-size limits.

The proposed alternative inverts the problem: instead of forcing the model to pick from the real vocabulary, ask a small, cheap model to invent a plausible category for the query with no taxonomy attached. The result is a fabricated path that doesn’t exist in your system—something like “Furniture / Living Room / Tables / Coffee.” That hallucination is then resolved to a real label by embedding every genuine classification (here with MiniLM) once, embedding the made-up one, and taking the nearest match by dot product, which lands on the correct “Coffee Tables & End Tables / Coffee Tables” node.

The payoff is cost and scale: the schema never travels to the model, dumb models suffice for the generation step, and the token-limit ceiling disappears. It’s a neat reframing of retrieval-style resolution—treat the LLM as an idea generator and let a local embedding index do the constraining, rather than trying to make the model obey the vocabulary directly.