RC RANDOM CHAOS

MicroLLM Lab runs seven models without touching the network

MicroLLM Lab runs seven 25M-360M parameter language models in the browser via WebGPU, on-device with no server, and benchmarks their speed and accuracy.

· 2 min read
MicroLLM Lab runs seven models without touching the network

MicroLLM Lab loads a language model of 25 to 360 million parameters into a browser tab, runs the inference on the local GPU through WebGPU, and never sends a prompt anywhere. Seven models are on offer. You click Load, the weights cache into the browser’s IndexedDB, and from there the tab chats on-device with no account, no API key, and no server behind it.

The size is what makes this work. The models are quantized to Q4, four bits per weight instead of the usual sixteen, which cuts the memory footprint by about 75%. A model north of 100 million parameters then fits in roughly 50 to 84 MB of browser memory, at what the lab calls near-lossless generation quality. WebGPU does the compute, issuing shaders to whatever the machine exposes: Metal on Apple Silicon, DirectX 12 on Windows, Vulkan elsewhere, all inside the browser with no native install.

A model in this range is an edge layer, built for specific tasks rather than broad trivia. It classifies a query, filters spam, or pulls intent out of a sentence, then decides whether an expensive cloud model is worth calling at all. The lab claims sub-10ms time-to-first-token, fast enough for autocomplete and real-time routing where a network round trip to a frontier API would dominate the latency budget.

The benchmark tab is the honest part. It scores on objective checks, regex matches and exact tokens, not on writing quality, and it reports speed and accuracy separately. Speed is tokens per second on sustained decode plus total suite wall time. Accuracy is the pass rate on the objective tests. A 135M model is allowed to fail those checks, and the lab says so directly: the failure is the measurement. The usual demo for a tiny model hides its weaknesses behind a cherry-picked prompt. Here the number is the number, and it stays on the machine that produced it.

If you run inference systems, the architecture is the interesting part, not the chat panel. A triage model living in the client removes a class of cost and exposure at once: no per-token bill, no prompt leaving the device, and no concurrency ceiling, because every user brings their own GPU. The trade is capability. These models route and classify; they are built for specific tasks rather than broad trivia. Benchmark them on the one task you intend to give them, on the hardware your users actually have, because WebGPU runs the inference on each client’s own GPU.

One operational detail: the weights persist in IndexedDB after the first load, so reopening the page is instant, but the model also sits in browser storage until it is cleared. On a shared machine that is a footprint to account for.

Share

Keep Reading

Latest on the Wire

Full wire →

New signal daily · RSS

Stay in the loop

New writing delivered when it's ready. No schedule, no spam.