The economics of LLM inference are collapsing — and local frontier AI is coming

The cost of running machine-learning inference is dropping by orders of magnitude per year, and the author argues this trend has enough independent tailwinds behind it to keep going. The through-line: within a year or two, LLMs become ambient infrastructure baked into every layer of computing rather than a standalone product, and within three to six years frontier-quality models run locally on commodity hardware. At that point the binding constraint on AI use shifts from raw token throughput to quality and access.

The case rests on several compounding gains rather than any single breakthrough. GPU power efficiency is doubling roughly every two years, an exponential curve unseen since Moore’s Law. Cost-per-task (not cost-per-token) is falling sharply as the price/quality Pareto frontier shifted about two orders of magnitude cheaper across 2026. Inference engines like vLLM, plus vendor stacks from NVIDIA and Intel, are posting 40%–140% efficiency and throughput gains purely from software, on fixed hardware. Architecturally, Mixture-of-Experts designs deliver the same benchmark quality at up to 7x fewer active parameters, though the savings favor hosted deployments since all experts must still sit in memory.

The most consequential shift for local AI is memory. Transformer models retain every input across every layer, demanding enormous RAM that AI firms have largely bought up. Mamba-based architectures instead keep a lossy running summary — essentially compaction built into the model — and Mamba-Transformer hybrids cut RAM needs by 5x or more, with models like Nemotron-H-47B holding very long contexts. Taken together, the piece frames a near future where inference is effectively too cheap to meter and capable models escape the datacenter.