WebLLM runs LLMs entirely in the browser via WebGPU — no server, OpenAI-compatible

WebLLM is an inference engine that executes large language models directly inside a web browser, tapping WebGPU for GPU acceleration so no backend or API server is involved. Because every token is generated on the user’s machine, the approach sidesteps the privacy and cost tradeoffs of routing prompts through a cloud provider while still delivering hardware-accelerated performance. It ships as an npm package (or a CDN import for quick prototyping) and is built as a companion to the MLC LLM project.

The engine mirrors the OpenAI API surface, so existing client code for chat completions, streaming, JSON-mode structured output, seeding, and logit-level control largely carries over — with the caveat that the model is loaded via a factory call rather than passed per request. Structured JSON generation is pushed into a WebAssembly layer for speed, and offloading to Web Workers or Service Workers keeps the UI responsive during inference. It also supports building Chrome extensions on top of the engine.

Out of the box it runs quantized builds of common open models including Llama 3, Phi 3, Gemma, Mistral, and Qwen, with a path to compiling custom models into MLC format. Models must be downloaded and cached on first use — a potentially heavy one-time cost — and WebLLM offers four cache backends (the Cache API by default, plus IndexedDB, OPFS, and an experimental cross-origin storage extension) to manage that footprint.